Although D catches lots of runtime errors,
programs tend to die silently if the following
code is used with huge text files:

chararr=cast(char[])read(bigfile);

If I remember correctly, the problem starts
with files of approx. 6MB+ in size. In some
cases the file will load, the program will
finish all operations, but will never return to
the command line (DmD 0.121, Win version).

I always had the perception that range and
stack overflow checking is enabled by default
and the according exceptions would be flagged.
Right or wrong?

"Bob W" <nospam@aol.com> wrote in message news:d49lvn$19h7$1@digitaldaemon.com...
> Although D catches lots of runtime errors,
> programs tend to die silently if the following
> code is used with huge text files:
>
> chararr=cast(char[])read(bigfile);
>
> If I remember correctly, the problem starts
> with files of approx. 6MB+ in size. In some
> cases the file will load, the program will
> finish all operations, but will never return to
> the command line (DmD 0.121, Win version).
>
> I always had the perception that range and
> stack overflow checking is enabled by default
> and the according exceptions would be flagged.
> Right or wrong?

It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.

For huge files, using std.stream or std.mmfile to read it is a better choice.

"Ben Hinkle" <ben.hinkle@gmail.com> wrote in message news:d49m8b$19lr$1@digitaldaemon.com...
>
> "Bob W" <nospam@aol.com> wrote in message news:d49lvn$19h7$1@digitaldaemon.com...
>> Although D catches lots of runtime errors,
>> programs tend to die silently if the following
>> code is used with huge text files:
>>
>> chararr=cast(char[])read(bigfile);
>>
>> If I remember correctly, the problem starts
>> with files of approx. 6MB+ in size. In some
>> cases the file will load, the program will
>> finish all operations, but will never return to
>> the command line (DmD 0.121, Win version).
>>
>> I always had the perception that range and
>> stack overflow checking is enabled by default
>> and the according exceptions would be flagged.
>> Right or wrong?
>
> It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.

Thanks for your reply, but I cannot help to believe
that there are some other things going wrong:

Given the most simplistic program:
read file to buffer, display buffer length, done.

Examples:
- File size 154,223,881: returns within the fraction of a sec.
- File size 158,923,657: waits forever (i.e. until my patience
  is exhausted and/or ^C is pressed).

(Before anyone asks: 1GB installed)

If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.

"Ben Hinkle" <ben.hinkle@gmail.com> wrote:

[...]
> It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op.
[...]

I remember that posts, but what is the purpose of a collect at exit anyway?

There is another point: I once tried to allocate all memory but did not succeed because no outOfMemoryException was thrown. Instead the program did not return.

I reported on that, but nobody seemed interested.

-manfred

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Manfred Nowak schrieb am Fri, 22 Apr 2005 03:39:05 +0000 (UTC):
> "Ben Hinkle" <ben.hinkle@gmail.com> wrote:
>
> [...]
>> It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op.
> [...]
>
> I remember that posts, but what is the purpose of a collect at exit anyway?
>
> There is another point: I once tried to allocate all memory but did not succeed because no outOfMemoryException was thrown. Instead the program did not return.

How did your try to "allocate all memory"?

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFCaHS53w+/yD4P9tIRAq7OAJ0VKdl78oinqzs5TDHlhTShgpuoegCfdv6/
OTfr+vjnpOF0UvzzxeXE1OM=
=n59e
-----END PGP SIGNATURE-----

Thomas Kuehne <thomas-dloop@kuehne.thisisspam.cn> wrote:

[...]
> How did your try to "allocate all memory"?

See bugs group.

-manfred

"Bob W" <nospam@aol.com> wrote in message news:d49qpp$1dcp$1@digitaldaemon.com...
>
> "Ben Hinkle" <ben.hinkle@gmail.com> wrote in message news:d49m8b$19lr$1@digitaldaemon.com...
>>
>> "Bob W" <nospam@aol.com> wrote in message news:d49lvn$19h7$1@digitaldaemon.com...
>>> Although D catches lots of runtime errors,
>>> programs tend to die silently if the following
>>> code is used with huge text files:
>>>
>>> chararr=cast(char[])read(bigfile);
>>>
>>> If I remember correctly, the problem starts
>>> with files of approx. 6MB+ in size. In some
>>> cases the file will load, the program will
>>> finish all operations, but will never return to
>>> the command line (DmD 0.121, Win version).
>>>
>>> I always had the perception that range and
>>> stack overflow checking is enabled by default
>>> and the according exceptions would be flagged.
>>> Right or wrong?
>>
>> It could be the giant gc collection going on at program exit. Try recompiling phobos with the function gc_term in phobos/internal/gc/gc.d set to a no-op. Currently it collects and given a large amount of memory I remember posts from long ago where the program could take a minute (maybe they or my memory were exagerating..) to exit while it runs though checking memory.
>
> Thanks for your reply, but I cannot help to believe
> that there are some other things going wrong:
>
> Given the most simplistic program:
> read file to buffer, display buffer length, done.
>
> Examples:
> - File size 154,223,881: returns within the fraction of a sec.
> - File size 158,923,657: waits forever (i.e. until my patience
>  is exhausted and/or ^C is pressed).
>
> (Before anyone asks: 1GB installed)
>
> If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.

You are probably right - one would think that if the GC were to blame it would be linear. I don't know what the cause is. Maybe you can try putting some printfs in internal/dmain2.d after your main returns and see where it gets stuck.

>>
>> If you have a good explanation why a 3% file size increase could possibly cause a maybe almost infinite delay increase, I'll do the Phobos recompilation. Otherwise I just patiently wait for further info.
>
> You are probably right - one would think that if the GC were to blame it would be linear. I don't know what the cause is. Maybe you can try putting some printfs in internal/dmain2.d after your main returns and see where it gets stuck.

I could but for now I will follow Walter's advice:
Stay away from std.file.read() for huge files.
He probably has had some info on these issues
before. And, who knows, maybe the read() is
already shining in DMD 0.122 or DMD 0.123?   ; )