Are we talking about executable size or sheer object file size?

Andrei

On 5/3/11 9:57 AM, Walter Bright wrote:
>
>
> On 5/3/2011 7:39 AM, Steve Schveighoffer wrote:
>> Dmd has steadily been making binaries more bloated for the last couple years (many times pointed out in the NG and there are several bugzilla reports).
>
> There are two causes of this:
>
> 1. Typeinfo getting larger, and people wanting more features in Typeinfo (like precise gc) so this will continue to increase.
>
> 2. Every phobos module imports and relies on every other one.
>
>
> To find out why .o files are large, run obj2asm on it. To find out why executables are large, compile with the -map switch and examine the linker .map file. <rant>AFAIK, I'm the only person who ever does this.</rant>
>
> Please, everyone. Once in a while, run obj2asm on the library module
> you're developing and have a look-see at what is coming out of the compiler.
>
>
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos

Both, although they aren't necessarily the same problem.

On 5/3/2011 10:18 AM, Andrei Alexandrescu wrote:
> Are we talking about executable size or sheer object file size?
>
> Andrei
>
> On 5/3/11 9:57 AM, Walter Bright wrote:
>>
>>
>> On 5/3/2011 7:39 AM, Steve Schveighoffer wrote:
>>> Dmd has steadily been making binaries more bloated for the last couple years (many times pointed out in the NG and there are several bugzilla reports).
>>
>> There are two causes of this:
>>
>> 1. Typeinfo getting larger, and people wanting more features in Typeinfo (like precise gc) so this will continue to increase.
>>
>> 2. Every phobos module imports and relies on every other one.
>>
>>
>> To find out why .o files are large, run obj2asm on it. To find out why executables are large, compile with the -map switch and examine the linker .map file. <rant>AFAIK, I'm the only person who ever does this.</rant>
>>
>> Please, everyone. Once in a while, run obj2asm on the library module
>> you're developing and have a look-see at what is coming out of the compiler.
>>
>>
>>
>> _______________________________________________
>> phobos mailing list
>> phobos at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/phobos
>
>

This is not a fair response to this problem.? When the code size triples (http://d.puremagic.com/issues/show_bug.cgi?id=2254) because you add the -lib switch, and adding unit tests adds to the size of the executable, even when unit tests are turned off (http://d.puremagic.com/issues/show_bug.cgi?id=5560) again, when adding the -lib switch, it's obvious to everyone (but you it seems) that the -lib switch is adding bloat.? Bug 5660 proves that beyond all reasonable doubt.

Whether it accounts for all the added bloat, I'm not sure.? obj2asm is a great tool, but I am not an assembly developer, I can't always decipher what's going on.? the -map switch only tells you what the compiler is outputting, it doesn't tell you if the output contains unnecessary bloat.

Please, at least agree to look at bug 5560, it's irrefutable proof that the -lib switch adds bloat, it cannot get any simpler than that.

-Steve




>________________________________
>From: Walter Bright <walter at digitalmars.com>
>To: Steve Schveighoffer <schveiguy at yahoo.com>; Discuss the phobos library for D <phobos at puremagic.com>
>Sent: Tuesday, May 3, 2011 12:57 PM
>Subject: large object files and binaries
>
>
>
>
>On 5/3/2011 7:39 AM, Steve Schveighoffer wrote: Dmd has steadily been making binaries more bloated for the last couple years (many times pointed out in the NG and there are several bugzilla reports).
>>
>There are two causes of this:
>
>1. Typeinfo getting larger, and people wanting more features in
    Typeinfo (like precise gc) so this will continue to increase.
>
>2. Every phobos module imports and relies on every other one.
>
>
>To find out why .o files are large, run obj2asm on it. To find out
    why executables are large, compile with the -map switch and examine
    the linker .map file. <rant>AFAIK, I'm the only person who
    ever does this.</rant>
>
>Please, everyone. Once in a while, run obj2asm on the library module
    you're developing and have a look-see at what is coming out of the
    compiler.
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20110503/50db3aa3/attachment.html>

Le 2011-05-03 ? 13:30, Steve Schveighoffer a ?crit :

> This is not a fair response to this problem.  When the code size triples (http://d.puremagic.com/issues/show_bug.cgi?id=2254) because you add the -lib switch, and adding unit tests adds to the size of the executable, even when unit tests are turned off (http://d.puremagic.com/issues/show_bug.cgi?id=5560) again, when adding the -lib switch, it's obvious to everyone (but you it seems) that the -lib switch is adding bloat.  Bug 5660 proves that beyond all reasonable doubt.
> 
> Whether it accounts for all the added bloat, I'm not sure.  obj2asm is a great tool, but I am not an assembly developer, I can't always decipher what's going on.  the -map switch only tells you what the compiler is outputting, it doesn't tell you if the output contains unnecessary bloat.
> 
> Please, at least agree to look at bug 5560, it's irrefutable proof that the -lib switch adds bloat, it cannot get any simpler than that.

Interesting. I might have an answer to that.

DMD emits code in a special "multiobj" mode when using the -lib switch. I had to disable this in my Objective-C-aware branch of the compiler because it didn't play well with Objective-C class definitions (maybe I could make it work, but I have more pressing things to do).

Now, I observe that Phobos compiled with my modified DMD makes 2501452 bytes, whereas the one that comes bundled with DMD makes 6646800 bytes... Granted, it's not exactly the same version of Phobos and druntime (I'm probably a little behind but with a few things added to my druntime branch), but the stunning difference leads me to believe "multiobj" might be the cause for this difference in size.

If someone wants to experiment, just open src/mars.c in DMD's source and change this line 842 from this:

	global.params.multiobj = 1;

to this:

	global.params.multiobj = 0;

I guess this will greatly reduce size of the library, although I have no idea whether it will affect the size of the final executable or not.


-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/

On 5/3/2011 1:32 PM, Michel Fortin wrote:
>
> DMD emits code in a special "multiobj" mode when using the -lib switch. I had to disable this in my Objective-C-aware branch of the compiler because it didn't play well with Objective-C class definitions (maybe I could make it work, but I have more pressing things to do).
>
> Now, I observe that Phobos compiled with my modified DMD makes 2501452 bytes, whereas the one that comes bundled with DMD makes 6646800 bytes... Granted, it's not exactly the same version of Phobos and druntime (I'm probably a little behind but with a few things added to my druntime branch), but the stunning difference leads me to believe "multiobj" might be the cause for this difference in size.

The size of libphobos.a has *nothing* to do with the size of an executable linked with Phobos.

Multiobj will increase the size of the library simply because there's a lot of bookkeeping overhead for each .o file - overhead that does not go into the executable.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20110503/d951f2e4/attachment.html>

On 5/3/2011 10:30 AM, Steve Schveighoffer wrote:
> This is not a fair response to this problem.  When the code size triples (http://d.puremagic.com/issues/show_bug.cgi?id=2254) because you add the -lib switch, and adding unit tests adds to the size of the executable, even when unit tests are turned off (http://d.puremagic.com/issues/show_bug.cgi?id=5560) again, when adding the -lib switch, it's obvious to everyone (but you it seems) that the -lib switch is adding bloat.  Bug 5660 proves that beyond all reasonable doubt.

Again, this is simply not relevant information. Each obj file has bookkeeping overhead required by the ELF file format. By looking at the object file size, you have ZERO information about the size of code/data making it into the executable.

>
> Whether it accounts for all the added bloat, I'm not sure.  obj2asm is a great tool, but I am not an assembly developer, I can't always decipher what's going on.

You do not need to know a single iota of assembler to understand where size is coming from in the output of obj2asm.

>   the -map switch only tells you what the compiler is outputting,

No, it doesn't. It tells you what symbols (i.e. functions) made it into the executable, and the size contributed by each function.

> it doesn't tell you if the output contains unnecessary bloat.

It does if the functions in the executable don't need to be there.

>
> Please, at least agree to look at bug 5560, it's irrefutable proof that the -lib switch adds bloat, it cannot get any simpler than that.

I'm afraid 5560 doesn't prove that. Without looking at the obj2asm contents of those .o files, you simply cannot conclude there is any bloat in there or if it is nothing more than the ELF bookkeeping.

>
> -Steve
>
>     --------------------------------------------------------------------------------
>     *From:* Walter Bright <walter at digitalmars.com>
>     *To:* Steve Schveighoffer <schveiguy at yahoo.com>; Discuss the phobos
>     library for D <phobos at puremagic.com>
>     *Sent:* Tuesday, May 3, 2011 12:57 PM
>     *Subject:* large object files and binaries
>
>
>
>     On 5/3/2011 7:39 AM, Steve Schveighoffer wrote:
>>     Dmd has steadily been making binaries more bloated for the last couple
>>     years (many times pointed out in the NG and there are several bugzilla
>>     reports).
>
>     There are two causes of this:
>
>     1. Typeinfo getting larger, and people wanting more features in Typeinfo
>     (like precise gc) so this will continue to increase.
>
>     2. Every phobos module imports and relies on every other one.
>
>
>     To find out why .o files are large, run obj2asm on it. To find out why
>     executables are large, compile with the -map switch and examine the linker
>     .map file. <rant>AFAIK, I'm the only person who ever does this.</rant>
>
>     Please, everyone. Once in a while, run obj2asm on the library module
>     you're developing and have a look-see at what is coming out of the compiler.
>
>
>
>
> _______________________________________________
> phobos mailing list
> phobos at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/phobos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20110503/599993eb/attachment.html>

I'm going to close 5560 because you are right, the -lib switch did not affect the final executable.? But I still don't understand the discrepancy, what bookkeeping is output for code that isn't generated?? I'm not trying for a witch hunt here, I just can't see how the compiler can change the size of an object file when essentially versioned out code is compiled.


However, there is definitely something wrong with -lib if 2254 is still valid.? I'll work on compiling phobos and druntime without -lib and see if I can repeat the difference in the final executable.


Also, can you explain how to use obj2asm to figure out if code is properly being generated without knowing assembly?? I should say, without knowing how to demangle names in your head as well.? I am not seeing how you do it without knowing these, every trip of mine through obj2asm land is painstakingly slow, and usually ends up with me giving up understanding what's going on.

-Steve




>________________________________
>From: Walter Bright <walter at digitalmars.com>
>To: Steve Schveighoffer <schveiguy at yahoo.com>; Discuss the phobos library for D <phobos at puremagic.com>
>Sent: Tuesday, May 3, 2011 6:22 PM
>Subject: Re: [phobos] large object files and binaries
>
>
>
>
>On 5/3/2011 10:30 AM, Steve Schveighoffer wrote:
>This is not a fair response to this problem.? When the code size triples (http://d.puremagic.com/issues/show_bug.cgi?id=2254) because you add the -lib switch, and adding unit tests adds to the size of the executable, even when unit tests are turned off (http://d.puremagic.com/issues/show_bug.cgi?id=5560) again, when adding the -lib switch, it's obvious to everyone (but you it seems) that the -lib switch is adding bloat.? Bug 5660 proves that beyond all reasonable doubt.
>Again, this is simply not relevant information. Each obj file has
    bookkeeping overhead required by the ELF file format. By looking at
    the object file size, you have ZERO information about the size of
    code/data making it into the executable.
>
>
>
>>
>>Whether it accounts for all the added bloat, I'm not sure.? obj2asm is a great tool, but I am not an assembly developer, I can't always decipher what's going on.
>You do not need to know a single iota of assembler to understand
    where size is coming from in the output of obj2asm.
>
>
>? the -map switch only tells you what the compiler is outputting, No, it doesn't. It tells you what symbols (i.e. functions) made it
    into the executable, and the size contributed by each function.
>
>
>it doesn't tell you if the output contains unnecessary bloat.
>It does if the functions in the executable don't need to be there.
>
>
>
>>
>>Please, at least agree to look at bug 5560, it's irrefutable proof that the -lib switch adds bloat, it cannot get any simpler than that.
>I'm afraid 5560 doesn't prove that. Without looking at the obj2asm
    contents of those .o files, you simply cannot conclude there is any
    bloat in there or if it is nothing more than the ELF bookkeeping.
>
>
>
>>
>>-Steve
>>
>>
>>
>>
>>>________________________________
>>>From: Walter Bright <walter at digitalmars.com>
>>>To: Steve Schveighoffer <schveiguy at yahoo.com>; Discuss the phobos library for D <phobos at puremagic.com>
>>>Sent: Tuesday, May 3, 2011 12:57 PM
>>>Subject: large object files and binaries
>>>
>>>
>>>
>>>
>>>On 5/3/2011 7:39 AM, Steve Schveighoffer wrote: Dmd has steadily been making binaries more bloated for the last couple years (many times pointed out in the NG and there are several bugzilla reports).
>>>>
>>>There are two causes of this:
>>>
>>>1. Typeinfo getting larger, and people wanting more
                  features in Typeinfo (like precise gc) so this will
                  continue to increase.
>>>
>>>2. Every phobos module imports and relies on every
                  other one.
>>>
>>>
>>>To find out why .o files are large, run obj2asm on it.
                  To find out why executables are large, compile with
                  the -map switch and examine the linker .map file.
                  <rant>AFAIK, I'm the only person who ever does
                  this.</rant>
>>>
>>>Please, everyone. Once in a while, run obj2asm on the
                  library module you're developing and have a look-see
                  at what is coming out of the compiler.
>>>
>>>
>>>
>>>
>>
_______________________________________________
phobos mailing list phobos at puremagic.com http://lists.puremagic.com/mailman/listinfo/phobos
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20110503/c704e36c/attachment-0001.html>

On 5/3/2011 4:14 PM, Steve Schveighoffer wrote:
>
> I'm going to close 5560 because you are right, the -lib switch did not affect the final executable.  But I still don't understand the discrepancy, what bookkeeping is output for code that isn't generated?

For example, there are all the definitions of the various sections.

>   I'm not trying for a witch hunt here, I just can't see how the compiler can
> change the size of an object file when essentially versioned out code is compiled.

To repeat myself, take a look at the obj2asm output. You can even diff one from another to see what changed.

>
> However, there is definitely something wrong with -lib if 2254 is still valid.  I'll work on compiling phobos and druntime without -lib and see if I can repeat the difference in the final executable.
>
> Also, can you explain how to use obj2asm to figure out if code is properly being generated without knowing assembly?

You can see the name of each function generated and how many bytes of code there are for it, for example. You can also see all the static data generated, such as:

       db  0,1,2,7,02Bh

which is pretty obvious.

>   I should say, without knowing how to demangle names in your head as well.  I
> am not seeing how you do it without knowing these, every trip of mine through
> obj2asm land is painstakingly slow, and usually ends up with me giving up
> understanding what's going on.
>

Feel free to ask questions here. It's probably a simple misunderstanding, because the output is much simpler than people presume it to be. Often, their presumption that it's complicated is what trips them up.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/phobos/attachments/20110503/24bc4887/attachment.html>

Le 2011-05-03 ? 18:13, Walter Bright a ?crit :

> On 5/3/2011 1:32 PM, Michel Fortin wrote:
>> 
>> DMD emits code in a special "multiobj" mode when using the -lib switch. I had to disable this in my Objective-C-aware branch of the compiler because it didn't play well with Objective-C class definitions (maybe I could make it work, but I have more pressing things to do).
>> 
>> Now, I observe that Phobos compiled with my modified DMD makes 2501452 bytes, whereas the one that comes bundled with DMD makes 6646800 bytes... Granted, it's not exactly the same version of Phobos and druntime (I'm probably a little behind but with a few things added to my druntime branch), but the stunning difference leads me to believe "multiobj" might be the cause for this difference in size.
> 
> The size of libphobos.a has *nothing* to do with the size of an executable linked with Phobos.
> 
> Multiobj will increase the size of the library simply because there's a lot of bookkeeping overhead for each .o file - overhead that does not go into the executable

Yeah, indeed. Creating many small object files instead of one is obviously going to add overhead.

I thought Steve said that the executable size went up when using -lib, which could have indicated a bug with multiobj.

-- 
Michel Fortin
michel.fortin at michelf.com
http://michelf.com/