I see functions marked with "nothrow", but inside they have:

     try {
       ...
     }
     catch (Exception e)
     {
            assert(0, "FracSec's constructor threw when it shouldn't have");
     }

This is redundant, and adds significant bloat. The compiler will statically check that the nothrow is enforced for you.

> On 5/2/2011 9:18 PM, Walter Bright wrote:
> > On 5/2/2011 1:14 PM, Jonathan M Davis wrote:
> >> The problem is far worse on Windows than on Linux - both in terms of the amount of memory eaten by dmd and because the Windows unit tests are all compiled together instead of separately, but dmd still eats too much memory on OSes other than Windows. FreeBSD would have the unit tests compiled separately like on Linux, so dmd shouldn't be running out of memory in the same way as occurs on Windows, but it could still be using a lot of memory, and if Walter's machine doesn't have a lot of it, and if FreeBSD tries to kill processes that use too much, then that could be the problem. On Windows, it very clearly states that dmd ran out of memory, but if the OS just kills the process, then you might not get such a clear message.
> >> 
> >> I've been hoping that Don's CTFE fixes would reduce dmd's memory footprint enough to get rid of issue 5454, but since they're buggy enough that Phobos' unit tests don't currently build, I have no idea how close they've gotten to fixing the problem.
> > 
> > On FreeBSD it's getting a "Killed: 9" message while running dmd after running for a looong time. Running under gdb shows it dying somewhere deep in malloc().
> 
> By deleting unittests one by one, I got it to pass dmd. Now if fails with a "Killed: 9" message from ld, the linker. Checking the object file for datetime.o yields a 4.6 Mb file (and that's without debug info turned on!). Clearly, reducing the memory consumption of dmd will not fix this problem.
> 
> Compiling std.datetime with -release -O gives a 441,000 byte object file.

I'm afraid that I don't understand what the problem is then. Can the linker not handle a 4.6 Mb object file? Granted, that's not a small file, but I wouldn't have thought that that would do in the linker.

And, of course, that raises the question as to why the object file is so large when compiling with unit tests. Part of it likely comes from the fact that the unit tests actually use the stuff in std.datetime, so all of the templated types and functions get instantiated, and they wouldn't when just compiling Phobos, unless something else in Phobos is using them. And maybe there's something in the unit tests which instantiates a bunch of extra templates. I don't know. Or did you compile with -release and -O _and_ -unittest? That would change things considerably, since that would imply a difference betwen release and non-release instead of a difference with regards to whether unit tests are compiled in.

Regardless, I don't know why the linker wouldn't like a 4.6 Mb object file. And while fixing dmd's memory consumption won't fix the problem, it might make it so that dmd actually finishes compiling and moves on to the linker rather than forcing you to delete unit tests. And whatever the problem may be, clearly it has something to do with the differences between your machine and the auto tester, since the auto tester is fine. So, regardless of dmd, whatever the linker's problem is, there's something about your machine that makes it fail while it succeeds on the auto tester.

And if all you want to do is to not have std.datetime fail on your machine, you can remove or comment out the line

    version = testStdDateTime;

(line #145, I believe) on your machine, and none of the std.datetime unit tests will be compiled in. It's already not defined on Windows specifically because dmd definitely runs out of memory compiling std.datetime on Windows when compiling all of Phobos' unit tests: http://d.puremagic.com/issues/show_bug.cgi?id=5454 (it does fine, if you compile std.datetime separately though).

- Jonathan M Davis

I suspect the 4,000 _assertPred instances add a lot to the memory consumption, as well as it taking a looong time to compile them all on my poor FreeBSD box. I know we've had this discussion before, but boring old assert() is quick to compile and memory efficient.

> I see functions marked with "nothrow", but inside they have:
> 
>      try {
>        ...
>      }
>      catch (Exception e)
>      {
>             assert(0, "FracSec's constructor threw when it shouldn't
> have"); }
> 
> This is redundant, and adds significant bloat. The compiler will statically check that the nothrow is enforced for you.

No, those _are_ required. One or more functions in the try block aren't nothrow. Logically, they _can't_ throw, but the compiler doesn't know that. So, I have to add the try-catch block so that the compiler won't scream about it. The assert(0) then ensures that the function fails if the logic is bad and the function _can_ throw.

- Jonathan M Davis

> I suspect the 4,000 _assertPred instances add a lot to the memory consumption, as well as it taking a looong time to compile them all on my poor FreeBSD box. I know we've had this discussion before, but boring old assert() is quick to compile and memory efficient.

Well, assert sucks in comparison to _assertPred, honestly, which is why it's used very little in std.datetime. I _have_ been rewriting the unit tests in std.datetime such that they use more loops and the like so that there will be fewer lines with _assertPred, but _assertPred is way better than straight assert, which is why I use it heavily. It makes debugging test failures much easier. But it takes time to rewrite the unit tests, and I only have so much, so it's taking a while. The fact that dmd currently fails to compile Phobos due to CTFE bugs doesn't help either.

- Jonathan M Davis

On 5/2/2011 11:06 PM, Jonathan M Davis wrote:
>
> I'm afraid that I don't understand what the problem is then. Can the linker not handle a 4.6 Mb object file? Granted, that's not a small file, but I wouldn't have thought that that would do in the linker.

It is an unusually large .o file, though. Maybe ld has some exponential behavior with the size of a .o file.

> And, of course, that raises the question as to why the object file is so large when compiling with unit tests.

(I made a mistake, it's 5Mb compiled with -unittest.) By replacing a few _assertPred's with assert, compiling and looking at the .o file size, each one consumes an extra 400 to over 1000 bytes. There are 4000 _assertPred's there, so that explains where the size comes from. There are 5 args that get pushed on the stack for each call to _assertPred, and one of those is a delegate (the lazy parameter).

I appreciate that using _assertPred saves you time and effort in development, but once it is developed it is a bit expensive to carry it around. I wonder if there's an easy way to swap it out with assert.

Sigh. Even obj2asm crashes with this .o file, but that's my fault.

Running "strings" on the .o for a non-unittest compile shows a lot of redundant strings that can be factored out to save space, such as "Standard Time" appearing 80 times or so.


>   Part of it likely comes from the fact that the
> unit tests actually use the stuff in std.datetime, so all of the templated
> types and functions get instantiated, and they wouldn't when just compiling
> Phobos, unless something else in Phobos is using them. And maybe there's
> something in the unit tests which instantiates a bunch of extra templates. I
> don't know. Or did you compile with -release and -O _and_ -unittest? That
> would change things considerably, since that would imply a difference betwen
> release and non-release instead of a difference with regards to whether unit
> tests are compiled in.
>
> Regardless, I don't know why the linker wouldn't like a 4.6 Mb object file. And while fixing dmd's memory consumption won't fix the problem, it might make it so that dmd actually finishes compiling and moves on to the linker rather than forcing you to delete unit tests. And whatever the problem may be, clearly it has something to do with the differences between your machine and the auto tester, since the auto tester is fine. So, regardless of dmd, whatever the linker's problem is, there's something about your machine that makes it fail while it succeeds on the auto tester.

My FreeBSD box is a 10 year old machine, with not much memory on it.

> And if all you want to do is to not have std.datetime fail on your machine, you can remove or comment out the line
>
>      version = testStdDateTime;
>
> (line #145, I believe) on your machine, and none of the std.datetime unit tests will be compiled in. It's already not defined on Windows specifically because dmd definitely runs out of memory compiling std.datetime on Windows when compiling all of Phobos' unit tests: http://d.puremagic.com/issues/show_bug.cgi?id=5454 (it does fine, if you compile std.datetime separately though).
>

For now, I'll disable the unittests for FreeBSD.

> On 5/2/2011 11:06 PM, Jonathan M Davis wrote:
> > I'm afraid that I don't understand what the problem is then. Can the linker not handle a 4.6 Mb object file? Granted, that's not a small file, but I wouldn't have thought that that would do in the linker.
> 
> It is an unusually large .o file, though. Maybe ld has some exponential behavior with the size of a .o file.
> 
> > And, of course, that raises the question as to why the object file is so large when compiling with unit tests.
> 
> (I made a mistake, it's 5Mb compiled with -unittest.) By replacing a few _assertPred's with assert, compiling and looking at the .o file size, each one consumes an extra 400 to over 1000 bytes. There are 4000 _assertPred's there, so that explains where the size comes from. There are 5 args that get pushed on the stack for each call to _assertPred, and one of those is a delegate (the lazy parameter).
> 
> I appreciate that using _assertPred saves you time and effort in development, but once it is developed it is a bit expensive to carry it around. I wonder if there's an easy way to swap it out with assert.

Simply swapping it out is not straightforward and would be annoying any time that any changes had to be made and something broke. Regardless, as I refactor the unit tests, the number of calls to _assertPred should decrease dramatically. Supposedly, Don was working on improving assert so that it provided decent output such that it was at least similar in benefit to _assertPred, but as far as I know, that hasn't gone anywhere yet. He's certainly busy with the CTFE at the moment in either case.

> Sigh. Even obj2asm crashes with this .o file, but that's my fault.
> 
> Running "strings" on the .o for a non-unittest compile shows a lot of redundant strings that can be factored out to save space, such as "Standard Time" appearing 80 times or so.

I don't believe that "Standard Time" is _ever_ in a string by itself. It's _always_ part of another string. So, replacing it would mean replacing a bunch of string literals with a smaller string literal plus a concatenation with an enum, const, or immutable string which held "Standard Time", and I'm not sure that that would help much. But maybe it would. It would arguably make those portions of the code less maintainable though. But maybe it would still be worth doing. I don't know quite what the repercussions would be either way.

- Jonathan M Davis

On 5/3/2011 1:11 AM, Jonathan M Davis wrote:
>
> Simply swapping it out is not straightforward and would be annoying any time that any changes had to be made and something broke.

I understand it would be annoying, but it's also annoying to not be able to run the unittests on Windows and FreeBSD. Considerable savings can be had by just getting rid of the __FILE__ parameter, as it's all in the same file.


>   Regardless, as I refactor
> the unit tests, the number of calls to _assertPred should decrease
> dramatically. Supposedly, Don was working on improving assert so that it
> provided decent output such that it was at least similar in benefit to
> _assertPred, but as far as I know, that hasn't gone anywhere yet. He's
> certainly busy with the CTFE at the moment in either case.

If it has those extra features, the size will inevitably come with it.

>> Sigh. Even obj2asm crashes with this .o file, but that's my fault.
>>
>> Running "strings" on the .o for a non-unittest compile shows a lot of redundant strings that can be factored out to save space, such as "Standard Time" appearing 80 times or so.
> I don't believe that "Standard Time" is _ever_ in a string by itself. It's _always_ part of another string. So, replacing it would mean replacing a bunch of string literals with a smaller string literal plus a concatenation with an enum, const, or immutable string which held "Standard Time", and I'm not sure that that would help much. But maybe it would. It would arguably make those portions of the code less maintainable though. But maybe it would still be worth doing. I don't know quite what the repercussions would be either way.
>

Looking at the code I see that you're doing a simple lookup table, but using explicit code instead. Replacing it with a two column table:

     string[][2] = [ ["AUS Central Standard Time", "Australia/Darwin"], ... ];

would run slower, but would occupy considerably less space than all those return statements. Using a table will also enable supporting a case insensitive lookup.

If it's any consolation, I just discovered that std.range won't unittest on FreeBSD with out of memory, too. I haven't investigated why.

Dmd has steadily been making binaries more bloated for the last couple years (many times pointed out in the NG and there are several bugzilla reports).


Also, I discovered dmd has a problem with unit tests affecting object size, even when they are turned off.? See a bug I created:

http://d.puremagic.com/issues/show_bug.cgi?id=5560


I'm not so sure dmd doesn't play a role in this problem.? There is definitely something going on with the compiler adding unnecessary bloat somewhere.? It seems to have to do with the -lib switch.

Has anyone tried compiling phobos without the -lib switch to see if it helps?


-Steve




>________________________________
>From: Walter Bright <walter at digitalmars.com>
>To: Discuss the phobos library for D <phobos at puremagic.com>
>Sent: Tuesday, May 3, 2011 1:43 AM
>Subject: Re: [phobos] FreeBSD 32 still fails unittests for std.datetime
>
>
>
>On 5/2/2011 9:18 PM, Walter Bright wrote:
>>
>>
>> On 5/2/2011 1:14 PM, Jonathan M Davis wrote:
>>>
>>> The problem is far worse on Windows than on Linux - both in terms of the amount of memory eaten by dmd and because the Windows unit tests are all compiled together instead of separately, but dmd still eats too much memory on OSes other than Windows. FreeBSD would have the unit tests compiled separately like on Linux, so dmd shouldn't be running out of memory in the same way as occurs on Windows, but it could still be using a lot of memory, and if Walter's machine doesn't have a lot of it, and if FreeBSD tries to kill processes that use too much, then that could be the problem. On Windows, it very clearly states that dmd ran out of memory, but if the OS just kills the process, then you might not get such a clear message.
>>>
>>> I've been hoping that Don's CTFE fixes would reduce dmd's memory footprint enough to get rid of issue 5454, but since they're buggy enough that Phobos' unit tests don't currently build, I have no idea how close they've gotten to fixing the problem.
>>>
>>
>> On FreeBSD it's getting a "Killed: 9" message while running dmd after running for a looong time. Running under gdb shows it dying somewhere deep in malloc().
>>
>
>By deleting unittests one by one, I got it to pass dmd. Now if fails with a "Killed: 9" message from ld, the linker. Checking the object file for datetime.o yields a 4.6 Mb file (and that's without debug info turned on!). Clearly, reducing the memory consumption of dmd will not fix this problem.
>
>Compiling std.datetime with -release -O gives a 441,000 byte object file.
>_______________________________________________
>phobos mailing list
>phobos at puremagic.com
>http://lists.puremagic.com/mailman/listinfo/phobos
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20110503/21e1cff6/attachment.html>

On 5/3/2011 7:39 AM, Steve Schveighoffer wrote:
> Dmd has steadily been making binaries more bloated for the last couple years (many times pointed out in the NG and there are several bugzilla reports).

There are two causes of this:

1. Typeinfo getting larger, and people wanting more features in Typeinfo (like precise gc) so this will continue to increase.

2. Every phobos module imports and relies on every other one.


To find out why .o files are large, run obj2asm on it. To find out why executables are large, compile with the -map switch and examine the linker .map file. <rant>AFAIK, I'm the only person who ever does this.</rant>

Please, everyone. Once in a while, run obj2asm on the library module you're developing and have a look-see at what is coming out of the compiler.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20110503/a4b3c18f/attachment.html>