Currently, D is using the same compilation model as C/C++, but I'm not convinced that it cannot be done better.

We have (at least):
- directories
- modules
- source files
- object files
- libraries
- binaries

The main reason for using the same compilation model as C/C++ is link compability, as I see it. But there is room for changes, while keeping link compability. There is a one-to-one correspondance between directories and modules. I suggest that we keep it this way. There is also a one-to-one correspondance between source files and object file. I suggest that is changed.

How about letting the compiler compile a whole module/directory at once, and emit a library. Object files within the library should be of much smaller smaller granularity than the source files, and there would be many more of them. Idially one object file per method or public data symbol. But it would not really be a concern using the compiler, because we - users of the compiler - would not mess with the object files, only libraries. The benefits would be:

- The compiler would be able to do module-wide optimizations.
- The linker (an existing, traditional linker) would be able to filter away
more unused code and data.
- When compiling a module, a source file needs only to be parsed once.

In the traditional model, when a change is made outside a module, it might happen that only a portion of the module needs to be recompiled. But in many cases, perhaps the most often cases, such changes means that the whole module needs to be recompiled. This leads me to think that we might as well treat the module as the translation unit, and gain the benefits above.

Comments?

Regards,
Martin M. Pedersen

D is designed so that an advanced compiler can compile the entire project in one go. For the time being it follows the traditional C/C++ development model because I have limited resources, and so take maximum advantage of existing tools.

"Martin M. Pedersen" <mmp@www.moeller-pedersen.dk> wrote in message news:b9uh9j$2fls$1@digitaldaemon.com...
> Currently, D is using the same compilation model as C/C++, but I'm not convinced that it cannot be done better.
>
> We have (at least):
> - directories
> - modules
> - source files
> - object files
> - libraries
> - binaries
>
> The main reason for using the same compilation model as C/C++ is link compability, as I see it. But there is room for changes, while keeping
link
> compability. There is a one-to-one correspondance between directories and modules. I suggest that we keep it this way. There is also a one-to-one correspondance between source files and object file. I suggest that is changed.
>
> How about letting the compiler compile a whole module/directory at once,
and
> emit a library. Object files within the library should be of much smaller smaller granularity than the source files, and there would be many more of them. Idially one object file per method or public data symbol. But it
would
> not really be a concern using the compiler, because we - users of the compiler - would not mess with the object files, only libraries. The benefits would be:
>
> - The compiler would be able to do module-wide optimizations.
> - The linker (an existing, traditional linker) would be able to filter
away
> more unused code and data.
> - When compiling a module, a source file needs only to be parsed once.
>
> In the traditional model, when a change is made outside a module, it might happen that only a portion of the module needs to be recompiled. But in
many
> cases, perhaps the most often cases, such changes means that the whole module needs to be recompiled. This leads me to think that we might as
well
> treat the module as the translation unit, and gain the benefits above.
>
> Comments?
>
> Regards,
> Martin M. Pedersen
>
>

I find no flaw in your arguments, and agree it would be an improvement.

Sean

"Martin M. Pedersen" <mmp@www.moeller-pedersen.dk> wrote in message news:b9uh9j$2fls$1@digitaldaemon.com...
> Currently, D is using the same compilation model as C/C++, but I'm not convinced that it cannot be done better.
>
> We have (at least):
> - directories
> - modules
> - source files
> - object files
> - libraries
> - binaries
>
> The main reason for using the same compilation model as C/C++ is link compability, as I see it. But there is room for changes, while keeping
link
> compability. There is a one-to-one correspondance between directories and modules. I suggest that we keep it this way. There is also a one-to-one correspondance between source files and object file. I suggest that is changed.
>
> How about letting the compiler compile a whole module/directory at once,
and
> emit a library. Object files within the library should be of much smaller smaller granularity than the source files, and there would be many more of them. Idially one object file per method or public data symbol. But it
would
> not really be a concern using the compiler, because we - users of the compiler - would not mess with the object files, only libraries. The benefits would be:
>
> - The compiler would be able to do module-wide optimizations.
> - The linker (an existing, traditional linker) would be able to filter
away
> more unused code and data.
> - When compiling a module, a source file needs only to be parsed once.
>
> In the traditional model, when a change is made outside a module, it might happen that only a portion of the module needs to be recompiled. But in
many
> cases, perhaps the most often cases, such changes means that the whole module needs to be recompiled. This leads me to think that we might as
well
> treat the module as the translation unit, and gain the benefits above.
>
> Comments?
>
> Regards,
> Martin M. Pedersen
>
>

I agree, esp. with the need to resolve the one-to-one relationship between source file (module) and object file.

One way to do this could be manually, by using a
  #pragma split
which could produce from a source file
  module.d:
  ...independent code, part 1
  #pragma split
  ...independent code, part 2
  #pragma split
  ...independent code, part 3
three separate object files (via a compiler "split" option)
  module_001.obj
  module_002.obj
  module_003.obj
for inclusion into libaries.

This would allow to reduce the footprint of D applications:

- embedded people won't consider D with its current 60K
  minimium exe size (they think in the range 1-8K).

- needing Win32 structs you will need to compile and link
  some windows.d file into your application. Its 500 init
  default structures add about 30K. This could be splitted
  into reasonable parts without creating 500 independent
  source files to have a space optimized interface.

- it would allow a more efficient reuse situation, because adding
  some code to a module would not autimatically mean to add
  to the footprint of any application using the module.

Of course it would be more beautiful when this effect could be transparent to the programmer, so that the splitted parts were produced as one granular object file that could be linked just the typical monolythic object files. But I think linkers won't support this.

"Martin M. Pedersen" wrote:
> 
> Currently, D is using the same compilation model as C/C++, but I'm not convinced that it cannot be done better.
> 
> We have (at least):
> - directories
> - modules
> - source files
> - object files
> - libraries
> - binaries
> 
> The main reason for using the same compilation model as C/C++ is link compability, as I see it. But there is room for changes, while keeping link compability. There is a one-to-one correspondance between directories and modules. I suggest that we keep it this way. There is also a one-to-one correspondance between source files and object file. I suggest that is changed.
> 
> How about letting the compiler compile a whole module/directory at once, and emit a library. Object files within the library should be of much smaller smaller granularity than the source files, and there would be many more of them. Idially one object file per method or public data symbol. But it would not really be a concern using the compiler, because we - users of the compiler - would not mess with the object files, only libraries. The benefits would be:
> 
> - The compiler would be able to do module-wide optimizations.
> - The linker (an existing, traditional linker) would be able to filter away
> more unused code and data.
> - When compiling a module, a source file needs only to be parsed once.
> 
> In the traditional model, when a change is made outside a module, it might happen that only a portion of the module needs to be recompiled. But in many cases, perhaps the most often cases, such changes means that the whole module needs to be recompiled. This leads me to think that we might as well treat the module as the translation unit, and gain the benefits above.
> 
> Comments?
> 
> Regards,
> Martin M. Pedersen

--
Helmut Leitner    leitner@hls.via.at Graz, Austria   www.hls-software.com

In article <b9uh9j$2fls$1@digitaldaemon.com>, Martin M. Pedersen says...
>
>Currently, D is using the same compilation model as C/C++, but I'm not convinced that it cannot be done better.

It is not really the same model as C. Look at any other language - it seems more similar to anything else than to C.

>We have (at least):
>- directories
are not really relevant

>- modules

one module ^= one source file ^= one object file.

>- source files
>- object files
>- libraries

libraries are not a part of compilation model - they are used as an agregate of compiled modules and possibly their source (or in the future possibly parsed source).

>- binaries


>The main reason for using the same compilation model as C/C++ is link compability, as I see it. But there is room for changes, while keeping link compability. There is a one-to-one correspondance between directories and modules. I suggest that we keep it this way. There is also a one-to-one correspondance between source files and object file. I suggest that is changed.

Where do you see correspondence of directories to modules? Keep many modules in one directory.

>How about letting the compiler compile a whole module/directory at once, and emit a library. Object files within the library should be of much smaller smaller granularity than the source files, and there would be many more of them. Idially one object file per method or public data symbol. But it would not really be a concern using the compiler, because we - users of the compiler - would not mess with the object files, only libraries. The benefits would be:
>
>- The compiler would be able to do module-wide optimizations.
>- The linker (an existing, traditional linker) would be able to filter away
>more unused code and data.
>- When compiling a module, a source file needs only to be parsed once.

The compiler does inter-module optimisations, as far as possible. For that, the complete parsed source of the application and used libraries is contained in memory during the project compilation. This causes the whole library code to be re-parsed once per project compilation, but not more often than that. Maybe Walter could implement dumping of the parse trees to disk - this would be a great and major optimisation.

>In the traditional model, when a change is made outside a module, it might happen that only a portion of the module needs to be recompiled. But in many cases, perhaps the most often cases, such changes means that the whole module needs to be recompiled. This leads me to think that we might as well treat the module as the translation unit, and gain the benefits above.

Modules are as long as a programmer could keep track of - a few dozen pages at most. The curent D compiler doesn't take any significant time to compile such an amount. GCC does, and hence it could be of relevance with it...

Object files need not be of smaller granularity, since:
- unittest/start-up code is common for a module, and must be included anyway
doesn't mater how much of the module you use;
- splitting a module apart into tiniest parts is a common misconception - it
shouldn't help. Any decent linker is able to eliminate all functions which are
not referenced anywhere. This means, methods of a class and functions taking
part in a unittest are never removed. All other unused functions are.

>Comments?

Seems you're not exactly sure what you're talking about.

-i.

--i
MIDICLUB

Hi,

> Where do you see correspondence of directories to modules? Keep many
modules in
> one directory.

Sorry, I has been away from D awhile. I was thinking more in line of Java's packages. I a larger D project, I would probably organize the modules such that there would be a main module per directory, the one you imported elsewhere, and a number of submodules. In such a case I was thinking of threating all the sources in the directory as one translation unit.

> Any decent linker is able to eliminate all functions which are not referenced anywhere.

Then I'm sorry that I don't have a decent linker, or don't know how to operate them efficiently. Given these sources...

=== main.c
#include <stdio.h>
extern const char* foo();
int main() {
    puts(foo());
    return 0;
}
=== foobar.c
extern const char* baz();
const char* bar() {
    return baz();
}
const char* foo() {
    return "foo";
}
=== baz.c
const char* baz()
{
    return "This should not be in the executable";
}
===

.. I'm able to find "This should not be in the executable" in the executable. I have tried this with the linker of MSVC6 and DMC.

> Seems you're not exactly sure what you're talking about.

If I was, I would not ask for comments :-)

Regards,
Martin M. Pedersen

midiclub@tiscali.de wrote:
> - splitting a module apart into tiniest parts is a common misconception - it shouldn't help. Any decent linker is able to eliminate all functions which are not referenced anywhere. This means, methods of a class and functions taking part in a unittest are never removed. All other unused functions are.

I often heard this but never came upon a system that really did this.

So please name a linker for Windows or Linux that shows this behaviour together with current D object modules.

--
Helmut Leitner    leitner@hls.via.at Graz, Austria   www.hls-software.com

Hello.

Martin M. Pedersen wrote:
> Then I'm sorry that I don't have a decent linker, or don't know how
> to operate them efficiently. Given these sources...
--- 8< --- >8 ---
> .. I'm able to find "This should not be in the executable" in the executable. I have tried this with the linker of MSVC6 and DMC.

This doesn't show much, since "This should not be ..." is data, not
code, and stored separately. However the code seems to be there too, i
checked it. :( I'm yet to try some other linkers. Maybe Borland or
Watcom? But from the times i worked with Delphi, yet version 2 or 3, i
can remember the linker stripping out functions even from debug
executables, midst in the actively used modules. This manifestated
itself in the debugger complaints when using real-time expression
evaluator: "This function has been eliminated by linker." This sometimes
gave me a clue what's wrong with my code - that it contains some silly
plug (a "bone") instead of real code - a function call - which was intended.

Sorry for my misconception - this probably comes from me reading too
much documentation containing hidden advertisements. :> But if such claims exist, there probably is some product which fulfills them...

>> Seems you're not exactly sure what you're talking about
> 
> If I was, I would not ask for comments :-)

It appears that i'm also not.

-i.

In article <3EC32D89.2F4CD480@chello.at>, Helmut Leitner says...
>
>I agree, esp. with the need to resolve the one-to-one relationship between source file (module) and object file.

I think that Walter's current mapping is fine. It doesn't prevent a D build-system from doing a global optimization of the whole "program" for release. Which would be nice for real-time or number crunching apps.

>
>One way to do this could be manually, by using a
>  #pragma split
>which could produce from a source file
>  module.d:
>  ...independent code, part 1
>  #pragma split
>  ...independent code, part 2
>  #pragma split
>  ...independent code, part 3
>three separate object files (via a compiler "split" option)
>  module_001.obj
>  module_002.obj
>  module_003.obj
>for inclusion into libaries.

yuk pragma stuff
>
>This would allow to reduce the footprint of D applications:
>
>- embedded people won't consider D with its current 60K
>  minimium exe size (they think in the range 1-8K).

I do embedded, it comes in all sizes, the folks working in the above size range do NOT use a 32 bit (or bigger) processor (requirement for D early on). Most everything I work on now is 32 bit with some 16 bit processors.

Reuse, project complexity (makefile structure), etc are usually more important. The D module appears to support this.

"Helmut Leitner" <helmut.leitner@chello.at> wrote in message news:3EC32D89.2F4CD480@chello.at...
> I agree, esp. with the need to resolve the one-to-one relationship between source file (module) and object file.
>
> One way to do this could be manually, by using a
>   #pragma split
> which could produce from a source file
>   module.d:
>   ...independent code, part 1
>   #pragma split
>   ...independent code, part 2
>   #pragma split
>   ...independent code, part 3
> three separate object files (via a compiler "split" option)
>   module_001.obj
>   module_002.obj
>   module_003.obj
> for inclusion into libaries.

The D compiler automatically generates COMDATs for each function in a module, so they are individually linked in. It's equivalent to doing the above.

> This would allow to reduce the footprint of D applications:
> - embedded people won't consider D with its current 60K
>   minimium exe size (they think in the range 1-8K).

The D footprint is about 24k larger than the equivalent C footprint.

> - needing Win32 structs you will need to compile and link
>   some windows.d file into your application. Its 500 init
>   default structures add about 30K. This could be splitted
>   into reasonable parts without creating 500 independent
>   source files to have a space optimized interface.

Actually, I need to remove the need to link in the struct inits for 0 initted structs.