Justin C Calvarese wrote:
[snip]
> Oh, I thought the .obj file included mentions of things that are needed, but not contained in a particular .obj. I thought that's why "Error 42: Symbol Undefined" will appear if I don't give the compiler enough source files.
> 
> If that's not right, that would be a serious flaw in my proposal.

Oh, you want the compiler to parse the .obj files to generate some extra stuff before it tries to link?
So basically that would be Walters "Put all the TypeInfo instantiations into one special module and import it everywhere you need TypeInfo", but automated by the compiler when it gets to the point of linking an executable?
I'm not sure if that's workable either since I'm pretty sure it'll need to parse the source files to generate the TypeInfo, at the very least for user-defined types. At that point it may not even have access to those source files, especially when someone is using a closed-source library (via .di header files). This is also a problem with what I thought you meant, by the way.

[snip]
>>> Perhaps, it'd be a question of "Is it worth the
>>> effort?".
>>
>> It'll be worth the effort when one of _your_ projects fail to compile because of it :P.
> 
> Well, of course, my plan is contingent upon my projects successfully compiling. ;)

Thought so :).

>> Like I've mentioned earlier: I'm pretty sure this problem would go away entirely if the compiler simply generated all TypeInfo used in the module. If that generates larger intermediate object files I'm okay with that. In fact, that was how I thought it worked until I started reading about this problem...
> 
> If that'd solve the problem, that'd be an improvement from the status quo.
> 
> But I had the understanding that there is a problem with the linker picking the TypeInfo from an arbitrary .obj (such as a large module that isn't needed for a particular program)? I'm afraid the linker might continue to choose an inappropriate TypeInfo. Or do you plan for all of the TypeInfo's to be unique, thus probably still bloating the .exe (but in a different way)?

I was kind of hoping the compiler wouldn't go looking for a new object file that includes a symbol it hasn't seen yet if it's present in the object file that needs it.
After seeing some more discussion on hash tables used in OMF linkers, I'm not sure if that's what would happen. It'd depend on how the linker is implemented, I guess.

>> [snip special linker discussion]
>>>> Though arguably the situation with DMD/Windows is already worse when
>>>> it comes to that, since almost nobody else uses OMF anymore...
>>>
>>> Right. We seem to be on our own when it comes to using OMF.
>>
>> Well, it seems OpenWatcom supports it. From what I've read here the linker doesn't like DMD object files though. Walter claims it's buggy. I don't know enough about OMF to say one way or the other.
> 
> Well, it doesn't really matter to me if DMD continues to use OMF if the format doesn't cause a bunch of bloat or other broken features.

Well, obviously it doesn't really matter if it works (and works well) :P.
Unfortunately, that doesn't currently seem to be the case...

> But I
> still wonder Walter needs to stay so close to the "official" format if
> DMC/DMD's OMF doesn't seems to be compatible with any other compiler.

Those bugs in OW might be fixed someday, or someone might re-implement OMF for the GNU toolchain (IIRC it was removed). Or someone else might want to implement a better linker for the Digital Mars compilers.
It's usually better to stick to published standards where they exist.

>> Yes, DMD/Linux uses ELF. It just calls ld (through gcc) to link instead of using optlink.
>>
>> I'm not sure if ld (or the mingw port of it) can use ELF to create Windows executables, but if it can that may be an option: just switch to ELF entirely and trash optlink. (this paragraph wasn't entirely serious, in case you hadn't noticed :P)
> 
> I suspect the option of ELF output would be welcomed by OMF's harshest critics. Not that I know anything about ELF.

Thinking about it, I seem to recall "PE operations on non-PE file" to be a common error when I was trying to link ELF files on Windows (cross-compiled). MinGW-ld didn't seem to like ELF object files.
That was when linking to ELF binaries though, not to Windows executables.

(I'm just going to interject here because WOW this thread is getting long; I can't even read the subject line anymore it's gone so far sideways...)

There's one suggestion I haven't seen yet, so I'll make it:

I assume from the discussion about segment-based linking that it's possible to pull out one particular section from an object file, and just link that into the executable.

So, why not make a small modification to OPTLINK such that if a switch
is thrown, and it encounters any missing symbol of the form
/_D11TypeInfo_.*/, then it will link only the segment that symbol is in.
 In other words, it does what it currently does in all cases except
where there's a TypeInfo involved; in which case it links *just* the
TypeInfo, not the whole object file.

This doesn't break compatibility with OMF or any other tool; it's simply an optimisation for reducing executable bloat in D programs.  This way, we don't need a new object format, or a whole new linker.

Or have I just got it all wrong? :P

	-- Daniel

-- 
Unlike Knuth, I have neither proven or tried the above; it may not even make sense.

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/

Daniel Keep wrote:
> (I'm just going to interject here because WOW this thread is getting
> long; I can't even read the subject line anymore it's gone so far
> sideways...)
> 
> There's one suggestion I haven't seen yet, so I'll make it:
> 
> I assume from the discussion about segment-based linking that it's
> possible to pull out one particular section from an object file, and
> just link that into the executable.
> 
> So, why not make a small modification to OPTLINK such that if a switch
> is thrown, and it encounters any missing symbol of the form
> /_D11TypeInfo_.*/, then it will link only the segment that symbol is in.
>  In other words, it does what it currently does in all cases except
> where there's a TypeInfo involved; in which case it links *just* the
> TypeInfo, not the whole object file.
> 
> This doesn't break compatibility with OMF or any other tool; it's simply
> an optimisation for reducing executable bloat in D programs.  This way,
> we don't need a new object format, or a whole new linker.
> 
> Or have I just got it all wrong? :P
> 
> 	-- Daniel
> 


On the face of it, that sounds like a reaonable solution. One would assume it would be legit to pull typeinfo from a segment instead ...

== Quote from Frits van Bommel (fvbommel@REMwOVExCAPSs.nl)'s article
> Justin C Calvarese wrote:
> [snip]
> > Oh, I thought the .obj file included mentions of things that are needed, but not contained in a particular .obj. I thought that's why "Error 42: Symbol Undefined" will appear if I don't give the compiler enough source files.
> >
> > If that's not right, that would be a serious flaw in my proposal.
> Oh, you want the compiler to parse the .obj files to generate some
> extra stuff before it tries to link?
> So basically that would be Walters "Put all the TypeInfo
> instantiations into one special module and import it everywhere you
> need TypeInfo", but automated by the compiler when it gets to the
> point of linking an executable?
> I'm not sure if that's workable either since I'm pretty sure it'll
> need to parse the source files to generate the TypeInfo, at the very
> least for user-defined types. At that point it may not even have
> access to those source files, especially when someone is using a
> closed-source library (via .di header files). This is also a problem
> with what I thought you meant, by the way.

Doh! I forgot that the compiler doesn't read the .obj/.lib files, but just hands them over to the linker.

Yeah, I'm seeing the big problems with my idea now. It's sounding like a lot more work than we could expect Walter or anyone else to undertake. There might still be an easy solution (e.g. having the compiler produce a text file that lists the needed TypeInfo's for each .obj that it outputs), but every solution seems to bring with it another risk (e.g. too many files floating around that have to be kept track of).

jcc7

kris wrote:
> Daniel Keep wrote:
>> (I'm just going to interject here because WOW this thread is getting
>> long; I can't even read the subject line anymore it's gone so far
>> sideways...)
>>
>> There's one suggestion I haven't seen yet, so I'll make it:
>>
>> I assume from the discussion about segment-based linking that it's
>> possible to pull out one particular section from an object file, and
>> just link that into the executable.
>>
>> So, why not make a small modification to OPTLINK such that if a switch
>> is thrown, and it encounters any missing symbol of the form
>> /_D11TypeInfo_.*/, then it will link only the segment that symbol is in.
>>  In other words, it does what it currently does in all cases except
>> where there's a TypeInfo involved; in which case it links *just* the
>> TypeInfo, not the whole object file.
>>
>> This doesn't break compatibility with OMF or any other tool; it's simply
>> an optimisation for reducing executable bloat in D programs.  This way,
>> we don't need a new object format, or a whole new linker.
>>
>> Or have I just got it all wrong? :P
>>
>>     -- Daniel
>>
> 
> 
> On the face of it, that sounds like a reaonable solution. One would assume it would be legit to pull typeinfo from a segment instead ...

Great idea if it's feasible... Would it then make sense that the switch be thrown by default for DMD?

== Quote from Daniel Keep (daniel.keep.lists@gmail.com)'s article
> (I'm just going to interject here because WOW this thread is getting long; I can't even read the subject line anymore it's gone so far sideways...)
>
> There's one suggestion I haven't seen yet, so I'll make it:
>
> I assume from the discussion about segment-based linking that it's possible to pull out one particular section from an object file, and just link that into the executable.
>
> So, why not make a small modification to OPTLINK such that if a switch is thrown, and it encounters any missing symbol of the form /_D11TypeInfo_.*/, then it will link only the segment that symbol is in.  In other words, it does what it currently does in all cases except where there's a TypeInfo involved; in which case it links *just* the TypeInfo, not the whole object file.
>
> This doesn't break compatibility with OMF or any other tool; it's
> simply
> an optimisation for reducing executable bloat in D programs.  This
> way, we don't need a new object format, or a whole new linker.
>
> Or have I just got it all wrong? :P
> 	-- Daniel

I don't know enough about how linkers work to know if OPTLINK can just include a TypeInfo segment like that, but it sounds like a good idea to me. I think it's more feasible than my idea was (since Frits helped me see some serious challenges with my idea).

And if it works, it should directly address Kris's problem.

jcc7

Carlos Smith wrote:
> gcc is not the right example.

It is for the Linux DMD.

Bill Baxter wrote:
> I see hangs occasionally even for small programs.  Even on single files compiled with dmd -run.  Every time it happens if I Ctrl-C kill it and run the same command again, everything is fine.  Frequency is maybe like 1 out of every 50 compiles.

I've never seen optlink crash except on known cases where there's a gigantic amount of static data. If you've got more conventional cases, please post a bug report with a reproducible case.

Sean Kelly wrote:
> Ideally, perhaps a linker could provide both options: link fast and potentially bloat the exe or link carefully (and slowly) for a lean exe. 

What is "link carefully"?

kris wrote:
> Walter Bright wrote:
>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.

Then the typeinfo for char[][] is being generated by another module. I suggest it would be helpful to find that module. Grep is a handy tool for finding it.

What let me to assume that the typeinfo for char[][] was *only* in core.obj was your statement that core got linked in regardless of where in the lib it was. This doesn't make sense in the light of it being also in some other .obj file.

I suggest identifying which .obj files have the typeinfo marked as extern, and which have it defined as a COMDAT.