I have a helloworld.d program. I compile it on both Win32 and on linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?

Turns out that D programs importing comprehensive Win32 D headers will wind up with executable space occupied for *all* constants and *all* struct initializers for the darned Win32 decls; where none of them are actually used.

That amounts to ~70KB of junk in the executable -- almost a full 100% increase in size beyond what it should be.

Does OptLink remove this? I've had no success with it. Walter? Can you help with this?

- Kris

kris wrote:
> I have a helloworld.d program. I compile it on both Win32 and on linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?
> 
> Turns out that D programs importing comprehensive Win32 D headers will wind up with executable space occupied for *all* constants and *all* struct initializers for the darned Win32 decls; where none of them are actually used.
> 
> That amounts to ~70KB of junk in the executable -- almost a full 100% increase in size beyond what it should be.
> 
> Does OptLink remove this? I've had no success with it. Walter? Can you help with this?
> 
> - Kris

Try using enums instead of const variables, they don't take up any space in the object file.

Walter Bright wrote:
> kris wrote:
>> I have a helloworld.d program. I compile it on both Win32 and on linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?
>>
>> Turns out that D programs importing comprehensive Win32 D headers will wind up with executable space occupied for *all* constants and *all* struct initializers for the darned Win32 decls; where none of them are actually used.
>>
>> That amounts to ~70KB of junk in the executable -- almost a full 100% increase in size beyond what it should be.
>>
>> Does OptLink remove this? I've had no success with it. Walter? Can you help with this?
> 
> Try using enums instead of const variables, they don't take up any space in the object file.

That shaved off maybe 30k.  I think the remaining 40k are mostly static struct initializers, all of which are unused in the final app.  As optlink is an optimizing linker, shouldn't it discard whatever is unused by the application?  Or does that only apply to certain classes of information?


Sean

Sean Kelly wrote:
> Walter Bright wrote:
>> Try using enums instead of const variables, they don't take up any space in the object file.
> 
> That shaved off maybe 30k.  I think the remaining 40k are mostly static struct initializers, all of which are unused in the final app.  As optlink is an optimizing linker, shouldn't it discard whatever is unused by the application?  Or does that only apply to certain classes of information?

I'm not sure about Windows, but I just tried it on Linux. It seems struct init data is stored in .rodata instead of some separate section[1]. That means if *any* read-only data from a module is referenced, all of its struct initializers are kept, presuming optlink works similarly to 'ld --gc-sections'.


[1]: Like .gnu.linkonce.*, where TypeInfo init data is stored (as well as functions).

Walter Bright wrote:
> kris wrote:
> 
>> I have a helloworld.d program. I compile it on both Win32 and on linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?
>>
>> Turns out that D programs importing comprehensive Win32 D headers will wind up with executable space occupied for *all* constants and *all* struct initializers for the darned Win32 decls; where none of them are actually used.
>>
>> That amounts to ~70KB of junk in the executable -- almost a full 100% increase in size beyond what it should be.
>>
>> Does OptLink remove this? I've had no success with it. Walter? Can you help with this?
>>
>> - Kris
> 
> 
> Try using enums instead of const variables, they don't take up any space in the object file.

Thanks, but that makes only a dent in the overhead.

One might assume that dmd and optlink have no facilities to remove unreferenced entities such as these? If so, doesn't this seem to be a notable weak point in the toolchain?

Why is this not an issue with C/C++ ?

1) C used true compile-time constants via #define. From your comment above, it would appear enum is the equivalent in D, rather than const int. Probably worth a note in the doc about that?

2) C does not have struct inititalizers.

Why has this not come to light before? Well, the Phobos Win32 headers are pretty darned slim compared with the real mcCoy. There's been a lot of clamour for 'real' win32 headers for years. Now that we have them, we're penalized for using 'em :p

Surely there must be some way for the linker to identify and discard unused entities?

Another approach would be to make the win32 headers be .di files, such that no obj files are generated. However, this breaks quickly whenever one uses a struct requiring an initializer (link errors).

Enum is only a small partial resolution here, and we need to complete the picture somehow. Any other suggestions, Walter? Can you tag the structs internally somehow to make the linker discard the unused ones? Can you explain exactly what's going on, such that others might be able to suggest something?

- Kris

kris wrote:
> Walter Bright wrote:
> 
>> kris wrote:
>>
>>> I have a helloworld.d program. I compile it on both Win32 and on linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?
>>>
>>> Turns out that D programs importing comprehensive Win32 D headers will wind up with executable space occupied for *all* constants and *all* struct initializers for the darned Win32 decls; where none of them are actually used.
>>>
>>> That amounts to ~70KB of junk in the executable -- almost a full 100% increase in size beyond what it should be.
>>>
>>> Does OptLink remove this? I've had no success with it. Walter? Can you help with this?
>>>
>>> - Kris
>>
>>
>>
>> Try using enums instead of const variables, they don't take up any space in the object file.
> 
> 
> Thanks, but that makes only a dent in the overhead.
> 
> One might assume that dmd and optlink have no facilities to remove unreferenced entities such as these? If so, doesn't this seem to be a notable weak point in the toolchain?
> 
> Why is this not an issue with C/C++ ?
> 
> 1) C used true compile-time constants via #define. From your comment above, it would appear enum is the equivalent in D, rather than const int. Probably worth a note in the doc about that?
> 
> 2) C does not have struct inititalizers.
> 
> Why has this not come to light before? Well, the Phobos Win32 headers are pretty darned slim compared with the real mcCoy. There's been a lot of clamour for 'real' win32 headers for years. Now that we have them, we're penalized for using 'em :p
> 
> Surely there must be some way for the linker to identify and discard unused entities?
> 
> Another approach would be to make the win32 headers be .di files, such that no obj files are generated. However, this breaks quickly whenever one uses a struct requiring an initializer (link errors).
> 
> Enum is only a small partial resolution here, and we need to complete the picture somehow. Any other suggestions, Walter? Can you tag the structs internally somehow to make the linker discard the unused ones? Can you explain exactly what's going on, such that others might be able to suggest something?

For example (off the top of my head to get things rolling), what would be the ramifications of placing each struct initializer into its own data-segment? Would the linker manage to remove the unused ones then? If so, could a compiler flag be used to enable that, or could it be done behind the scenes?

Other suggestions?

> Walter Bright wrote:
>>   Try using enums instead of const variables, they don't take up any space in the object file.
>
> Thanks, but that makes only a dent in the overhead.


Actually using enums instead of consts have a very nice side effect.
Consider this declaration from win32:

HANDLE CreateFileW(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE);

Now, which invocation is correct?

CreateFile("a", OPEN_EXISTING, FILE_SHARE_READ, null, GENERIC_READ ...)
or
CreateFile("a", OPEN_EXISTING, GENERIC_READ, null, FILE_SHARE_READ ...)
or even
CreateFile("a", GENERIC_READ, OPEN_EXISTING, null, FILE_SHARE_READ ...)?

The compiler happily accepts all of them.

Using enums, we have:

enum FILE_SHARE {
  READ  = 1,
  WRITE = 2,
  BOTH  = READ | WRITE
}

enum DISPOSITION {
  CREATE_ALWAYS = 1,
  OPEN_EXISTING = 2
}

void CreateFile(char[] fileName, DISPOSITION disposition, FILE_SHARE share) {}

int main() {
  // compiler allows unknown flags
  CreateFile("filename", cast(DISPOSITION)4, FILE_SHARE.READ);

  // when we swap the arguments by mistake
  CreateFile("filename", FILE_SHARE.READ, DISPOSITION.CREATE_ALWAYS);

  return 0;
}

This way if I make an error the compiler will very helpfully tell me
what exactly is happening. Consider: CreateFont has 14 parameters,
if the IDE does not help me i'd make an error when I use it, trust me

On the other hand, when a new version of Windows come with expanded options
I can just use the new numbers without breaking anything while the d package is
updated.

The dotnet framework uses enums and I really like the idea.

Regards,
Todor

Walter Bright wrote:
> kris wrote:
>> I have a helloworld.d program. I compile it on both Win32 and on linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?
>>
>> Turns out that D programs importing comprehensive Win32 D headers will wind up with executable space occupied for *all* constants and *all* struct initializers for the darned Win32 decls; where none of them are actually used.
>>
>> That amounts to ~70KB of junk in the executable -- almost a full 100% increase in size beyond what it should be.
>>
>> Does OptLink remove this? I've had no success with it. Walter? Can you help with this?
> 
> Try using enums instead of const variables, they don't take up any space in the object file.

For reference, the "hello world" app test was compiled with Bud, which basically just recursively compiled all .d files and links the objects together.  However, I just tried creating a library out of the Win32 modules and renamed them to .di files to keep Bud from building them, and the resulting app was 90k instead of 144k.  I don't suppose you can explain how linking a lib and linking individual object files can have such glaringly different effects?  I'm going to try a manual build without Bud and see if the app is 144k now too.


Sean

Todor Totev wrote:
>> Walter Bright wrote:
>>
>>>   Try using enums instead of const variables, they don't take up any  space in the object file.
>>
>>
>> Thanks, but that makes only a dent in the overhead.
> 
> 
> 
> Actually using enums instead of consts have a very nice side effect.
> Consider this declaration from win32:
> 
> HANDLE CreateFileW(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD,  DWORD, HANDLE);
> 
> Now, which invocation is correct?
> 
> CreateFile("a", OPEN_EXISTING, FILE_SHARE_READ, null, GENERIC_READ ...)
> or
> CreateFile("a", OPEN_EXISTING, GENERIC_READ, null, FILE_SHARE_READ ...)
> or even
> CreateFile("a", GENERIC_READ, OPEN_EXISTING, null, FILE_SHARE_READ ...)?
> 
> The compiler happily accepts all of them.
> 
> Using enums, we have:
> 
> enum FILE_SHARE {
>   READ  = 1,
>   WRITE = 2,
>   BOTH  = READ | WRITE
> }
> 
> enum DISPOSITION {
>   CREATE_ALWAYS = 1,
>   OPEN_EXISTING = 2
> }
> 
> void CreateFile(char[] fileName, DISPOSITION disposition, FILE_SHARE  share) {}
> 
> int main() {
>   // compiler allows unknown flags
>   CreateFile("filename", cast(DISPOSITION)4, FILE_SHARE.READ);
> 
>   // when we swap the arguments by mistake
>   CreateFile("filename", FILE_SHARE.READ, DISPOSITION.CREATE_ALWAYS);
> 
>   return 0;
> }
> 
> This way if I make an error the compiler will very helpfully tell me
> what exactly is happening. Consider: CreateFont has 14 parameters,
> if the IDE does not help me i'd make an error when I use it, trust me
> 
> On the other hand, when a new version of Windows come with expanded options
> I can just use the new numbers without breaking anything while the d  package is
> updated.
> 
> The dotnet framework uses enums and I really like the idea.
> 
> Regards,
> Todor


Certainly, and I think you'll find many who agree, along with some who can't bear to type a few extra characters :)

But that's a different type of issue than the one we're facing. I wish you had a fix for that one :p

Sean Kelly wrote:
> Walter Bright wrote:
>> kris wrote:
>>> I have a helloworld.d program. I compile it on both Win32 and on linux. On linux, the executable is 72KB whereas on Win32 it is 141KB. Why is this?
>>>
>>> Turns out that D programs importing comprehensive Win32 D headers will wind up with executable space occupied for *all* constants and *all* struct initializers for the darned Win32 decls; where none of them are actually used.
>>>
>>> That amounts to ~70KB of junk in the executable -- almost a full 100% increase in size beyond what it should be.
>>>
>>> Does OptLink remove this? I've had no success with it. Walter? Can you help with this?
>>
>> Try using enums instead of const variables, they don't take up any space in the object file.
> 
> For reference, the "hello world" app test was compiled with Bud, which basically just recursively compiled all .d files and links the objects together.  However, I just tried creating a library out of the Win32 modules and renamed them to .di files to keep Bud from building them, and the resulting app was 90k instead of 144k.  I don't suppose you can explain how linking a lib and linking individual object files can have such glaringly different effects?  I'm going to try a manual build without Bud and see if the app is 144k now too.

I'm beginning to suspect a compiler/linker bug.  Here is my first test supplying all modules on one line:

--------------------------------------------------------------------------------

C:\code\src\d\test\appsize>dmd -inline -release -O hello.d \code\import\tango\tango\io\Console.d \code\import\tango\tango\sys\Common.d \code\import\tango\tango\sys\win32\Types.d \code\import\tango\tango\sys\win32\Common.d \code\import\tango\tango\io\Buffer.d \code\import\tango\tango\io\Exception.d \code\import\tango\tango\io\model\IBuffer.d \code\import\tango\tango\io\model\IConduit.d \code\import\tango\tango\io\DeviceConduit.d \code\import\tango\tango\io\Conduit.d

c:\bin\dmd\bin\..\..\dm\bin\link.exe hello+Console+Common+Types+Common+Buffer+Ex
ception+IBuffer+IConduit+DeviceConduit+Conduit,,,user32+kernel32/noi;

C:\code\src\d\test>dir hello.exe
 Volume in drive C is System
 Volume Serial Number is 58B0-B8C5

 Directory of C:\code\src\d\test\appsize

01/23/2007  12:05 PM           141,852 hello.exe

--------------------------------------------------------------------------------

And my second test compiling all modules separately and then linking into an app:

--------------------------------------------------------------------------------

dmd -inline -release -O -c hello.d
dmd -inline -release -O -c \code\import\tango\tango\io\Console.d
dmd -inline -release -O -c \code\import\tango\tango\sys\Common.d
dmd -inline -release -O -c \code\import\tango\tango\sys\win32\Types.d
dmd -inline -release -O -c \code\import\tango\tango\sys\win32\Common.d -ofCommonWin32.obj
dmd -inline -release -O -c \code\import\tango\tango\io\Buffer.d
dmd -inline -release -O -c \code\import\tango\tango\io\Exception.d
dmd -inline -release -O -c \code\import\tango\tango\io\model\IBuffer.d
dmd -inline -release -O -c \code\import\tango\tango\io\model\IConduit.d
dmd -inline -release -O -c \code\import\tango\tango\io\DeviceConduit.d
dmd -inline -release -O -c \code\import\tango\tango\io\Conduit.d

C:\code\src\d\test\appsize>dmd hello *.obj
c:\bin\dmd\bin\..\..\dm\bin\link.exe hello+*,,,user32+kernel32/noi;

C:\code\src\d\test\appsize>dir hello.exe
 Volume in drive C is System
 Volume Serial Number is 58B0-B8C5

 Directory of C:\code\src\d\test\appsize

01/23/2007  12:13 PM           141,340 hello.exe

--------------------------------------------------------------------------------

And now my third test where tango/sys/win32/*.* is compiled into a library (win32.lib) and the lib is linked instead of the module object files:

--------------------------------------------------------------------------------

C:\code\src\d\test\appsize>lib -c -n Win32.lib CommonWin32.obj Types.obj
Digital Mars Librarian Version 8.00n
Copyright (C) Digital Mars 2000-2002 All Rights Reserved www.digitalmars.com
Digital Mars Librarian complete.

C:\code\src\d\test\appsize>del CommonWin32.obj Types.obj

C:\code\src\d\test\appsize>dmd hello *.obj Win32.lib
c:\bin\dmd\bin\..\..\dm\bin\link.exe hello+*,,,Win32.lib+user32+kernel32/noi;

C:\code\src\d\test\appsize>dir hello.exe
 Volume in drive C is System
 Volume Serial Number is 58B0-B8C5

 Directory of C:\code\src\d\test\appsize

01/23/2007  12:20 PM            90,140 hello.exe

--------------------------------------------------------------------------------

As you can see, simply putting the Win32 object files into a library prior to linking reduced the application size by 51,100 bytes, but separate compilation had the same (bad) result as compiling all modules on one line.  I grant that 51k may not be a huge issue for the average Windows app, but it could be a killer for handheld (WinCE) applications.  Could you please explain the size difference if this is indeed not a bug?  I don't mind making these files .di files and adding a pragma(lib, "Win32.lib"), but this seems like something that must be addressed in a more general manner at some point.


Sean