Walter Bright wrote:
> Frits van Bommel wrote:
> 
>> If so, the obvious workaround would be to tell it to only use one thread.
>> If not, why does multicore matter?
> 
> The way mutexes are done for multi core is different than for single core. I think optlink multithreads correctly only for single core machines.

Assuming you're talking about cores that don't share a cache (ie. "real" SMP) then it's possible optlink id just doing its own synchronization by manipulating variables.  Fixing it might be as easy as sticking a lock prefix in the right locations (long shot, I know).


Sean

On Fri, 23 Feb 2007 13:54:33 -0800, Walter Bright wrote:

> John Reimer wrote:
>> optlink may just be the bane for D acceptance. And Tango gets the pitiful opportunity of demonstrating why D is NOT ready for prime-time in the commercial realm: the DM support tools it relies on are bogged down in the past, reflecting D's lopsided existance on yet another level: a strong language relying on a fragile, outdated, and poorly fit tool set.
> 
> Linux's ld exhibits the same behavior. Try compiling the 3 files here according to the instructions below, and try different orderings adding the object files to the librarian (ar). The same behavior as lib/optlink results (using nm to see what symbols are placed into the resulting executable).
> 
> ------------------- test.d --------------------
> import b;       // a is not imported
> 
> void foo(...) { }
> 
> void test(char[][] s)
> {
>      bbb2();
>      foo(s);
> }
> 
> void main()
> {
> }
> --------------------- a.d -------------------
> void xxx(...) { }
> 
> void aaa(char[][] s)
> {
>      xxx(s);
> }
> 
> void aaa2()             // never referenced nor imported 'bloat'
> {
> }
> -------------------- b.d ---------------------
> void yyy(...) { }
> 
> void bbb(char[][] s)
> {
>      yyy(s);
> }
> 
> void bbb2()
> {
> }
> -------------------- build 1 -------------------
> dmd -c a.d test.d
> dmd -c b.d
> rm foo.a
> ar -r foo.a a.o b.o     <= a comes before b
> dmd test.o foo.a
> nm test >log		<= aaa2() appears in executable
> -------------------- build 2 -------------------
> dmd -c a.d test.d
> dmd -c b.d
> rm foo.a
> ar -r foo.a b.o a.o     <= b comes before a
> dmd test.o foo.a
> nm test >log		<= aaa2() does not appear in executable
> ---------------------------------------------


True, you are correct about the same error being represented here. This proves only that ld/ar and optlink/lib are on equal footing in this specific case. optlink/lib/OMF however have so many problems in other aspects that my opinion stands.  (one simple example: think 64-bit... dmd tools have no footing)

Concerning the problem at hand:

D is not C.  Resolving symbols in C era was part of the business and pretty much cause/effect.  Not so with D and its hidden implementation details and duplicate symbols resulting from implementation of language features.

That's why D is different; that's why jerry-wrigging old tools to a new
language and expecting developers to rely on these tools and fix
problems like they are C programmers is totally absurd. Even expert D users
are going to be confounded.  New users will be more lost than if they were
using C. That's bad publicity for D, no two ways about it.

What your example above actually points out is the specific weakness
in D's TypeInfo implementation such that linker/librarians are rendered
incapable of making sane choices.  Maybe it's not the linkers and
librarians fault since they are too stupid to really figure a resolution?
If not, then it must be the implementation's fault.

Two options?  Create new a new object file format, a new linker, and a new librarian to suite D's evolving feature set.  Everyone will yell "overkill", "that's a nasty amount of work", "I'm not gonna do it!", and "How are we going to support C?"

If we /must/ go a jerry-rigging to old tools in the name of keeping things simple (and keeping the faith with C), then perhaps the better option is to rethink how TypeInfo is implemented and mold it to fit sanely into the containers and tools available. Like I said, to expect programmers to troubleshoot hidden operations -- how linker and librarian operate with D symbols that are practically invisible from the programmers perspective -- is absolutely nasty and mean.  :)

This /really/ needs to be addressed.  Let's stop
saying that the programmer needs to learn to deal with this.  This is
really just a way of ignoring a show-stopping issue for D.  D cannot afford
to turn it's back on it anymore (as it has for the last few years).  D has
to be progressive in more than just language features to make an impact in
this world.

-JJR

Sean Kelly wrote:
> Frits van Bommel wrote:
>> Sean Kelly wrote:
>>> That makes complete sense.  It's irritating that such matches pull in the entire object, but at least the process is straightforward and logical.
>>
>> Of course, if matches didn't pull in the entire object then static constructors/destructors, unittests and exceptions wouldn't work without some way to flag specific sections as "always pull this in if anything (or even better: if _certain things_) are pulled in" or "when this is pulled in, also pull in X and Y even though they're not referenced" or something similar...
> 
> Hm... so how does segment-level linking work at all?

Well, ld has a switch called --gc-sections, which basically... wait for it... garbage collects the sections. :)
From what I understand, this currently breaks DMD exception handling on Linux, since nothing explicitly refers to the sections containing the data.
C++ exceptions work fine AFAIK, presumably because the default linker script[1] explicitly tells ld to keep .eh_frame sections even if unreferenced. It does the same for quite some other sections, including .ctors and .dtors (which DMD uses to set up the linked module list).

So presumably it first selects the objects to be linked in and then uses the program entry point and those sections as the "root pointers" for the --gc-sections switch if used.

This would mean that if a module gets pulled in, its .ctors section (if any) gets kept, and that references (indirectly) the static constructors, destructors and unit tests. So that makes me think ld may have the same issue as optlink, since Walter has shown that without --gc-sections ld also pulls in whole object files (even if the corresponding modules are not necessarily imported by the program). Unless ld somehow handles libraries in a smarter manner...


[1] use 'ld --verbose' to see it.

Walter Bright wrote:
> Bill Baxter wrote:
>> I see hangs occasionally even for small programs.  Even on single files compiled with dmd -run.  Every time it happens if I Ctrl-C kill it and run the same command again, everything is fine.  Frequency is maybe like 1 out of every 50 compiles.
> 
> I've never seen optlink crash except on known cases where there's a gigantic amount of static data. If you've got more conventional cases, please post a bug report with a reproducible case.

Hmm, well it's not crashing just hanging.  And it may not be optlink but dmd.  I do all my work on an external usb/firewire drive so that could be an issue (like optlink waiting for a file lock to be released?). Also it doesn't happen very often.  Maybe 1 out of 50 is too high an estimate.  More like once every few days (and I recompile things a lot).

It always works if I Ctrl-C and try again, so a repro will be difficult.

Other possibilities -- I also started using a cmd wrapper called "Console" about the same time as getting into D, so maybe that could also be related I suppose. [http://sourceforge.net/projects/console/]. It is a bit flaky at times.

In short it's really not a big issue.  Just something I see occasionally.  I posted it mostly to see if anyone else was silently seeing the same sorts of things.  If no-one else has seen this then it's most likely due to something in my particular setup.

--bb

Walter Bright wrote:
> Walter Bright wrote:
>> Bill Baxter wrote:
>>> I see hangs occasionally even for small programs.  Even on single files compiled with dmd -run.  Every time it happens if I Ctrl-C kill it and run the same command again, everything is fine.  Frequency is maybe like 1 out of every 50 compiles.
>>
>> I've never seen optlink crash except on known cases where there's a gigantic amount of static data. If you've got more conventional cases, please post a bug report with a reproducible case.
> 
> I'd forgotten, there is a problem with optlink running on multicore machines. There's supposed to be a way to tell Windows to run an exe using only one core, but I can't think of it at the moment.

Hmm, one of my machines is core duo, so that could maybe be it.

--bb

Bill Baxter wrote:
> I see hangs occasionally even for small programs.  Even on single files compiled with dmd -run.  Every time it happens if I Ctrl-C kill it and run the same command again, everything is fine.  Frequency is maybe like 1 out of every 50 compiles.

I got that too! Same numbers, 1:50. AMD dual core.

L.

I want to point out also that there /is/ a way to partially side-step this issue, believe it or not:

Use GDC and build Tango or Phobos as a shared library (on non-win32 systems naturally). Win32 dlls are painfully limited in this regard so it's a no-go there, and dmd (I believe) still doesn't support shared libs on linux.

Suddenly, the problem of fat binaries and phantom dependencies is
neatly abstracted away... well, in a matter of speaking only since the
issue still exists, I'm sure, in some sense. But seeing a Tango or Phobos
as a shared library tends to make us all have warm, fuzzy feelings about
the whole thing. :)

When alls said and done, though... probably the equivalent amount of object code is loaded into memory (phantom objects included), except that now the library is shared by D programs instead.

-JJR

Bill Baxter wrote:
> Walter Bright wrote:
>> Bill Baxter wrote:
>>> I see hangs occasionally even for small programs.  Even on single files compiled with dmd -run.  Every time it happens if I Ctrl-C kill it and run the same command again, everything is fine.  Frequency is maybe like 1 out of every 50 compiles.
>>
>> I've never seen optlink crash except on known cases where there's a gigantic amount of static data. If you've got more conventional cases, please post a bug report with a reproducible case.
> 
> Hmm, well it's not crashing just hanging.  And it may not be optlink but dmd.  I do all my work on an external usb/firewire drive so that could be an issue (like optlink waiting for a file lock to be released?). Also it doesn't happen very often.  Maybe 1 out of 50 is too high an estimate.  More like once every few days (and I recompile things a lot).
> 
> It always works if I Ctrl-C and try again, so a repro will be difficult.
> 
> Other possibilities -- I also started using a cmd wrapper called "Console" about the same time as getting into D, so maybe that could also be related I suppose. [http://sourceforge.net/projects/console/]. It is a bit flaky at times.
> 
> In short it's really not a big issue.  Just something I see occasionally.  I posted it mostly to see if anyone else was silently seeing the same sorts of things.  If no-one else has seen this then it's most likely due to something in my particular setup.
> 
> --bb

I've seen it before compiling the wxD samples.  I'm running a Pentium 4 hyperthreaded.  As you say, it's a hang, it's occasional, but I wouldn't say 1 in 50--it bites me just about every time I compile all the wxD samples, so more like 1 in 15 for me.

-Dave

Walter Bright wrote:
> kris wrote:
> 
>> Walter Bright wrote:
>>
>>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
>>
>> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.

After taking a much needed break from this, I'm having another bash at it. The locale package highlighted in the last bout has been through a number of changes, and thus the behaviour will now be somewhat different than before.

For a refresher on this issue, here's an overview: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=49257

The upshot is that reordering the content as presented to the librarian now has an effect on the resultant binary size. This tells me two things:

1) there were more than just the one compiler-generated unresolved symbol in the first case, though I did not spot it after many painstaking hours of frustrating effort. In fact, there may well have been a number of vaguely intertwined dependencies spread throughout the library, due entirely to these compiler-generated symbols.

2) there is no feasible manner in which a developer can control how lib contents are linked while the compiler continues to generate duplicate symbols under the covers. The fact that it does this on a regular basis simply serves to highlight a critical problem.


> Then the typeinfo for char[][] is being generated by another module. I
> suggest it would be helpful to find that module. Grep is a handy tool
> for finding it.

What would be the point? These symbols are compiler generated, and exist in a variety of places because of that fact alone. Lest we forget, we're not even talking about just one symbolic name ~ the compiler generates duplicate symbols for any typeinfo that does not match the pre-packaged list (such as char[][]). The end result is a potential rats-nest of duplicate symbols, creating a maze of intricate and fragile dependencies across the lib itself.

To put things into perspective, I have absolutely no way of knowing what the real dependencies within this library actually are. As such, I'd have to describe the situation as being "out of control". This is wholly due to the duplicate compiler-generated symbols.

kris wrote:
> Walter Bright wrote:
> 
>> kris wrote:
>>
>>> Walter Bright wrote:
>>>
>>>> Sure, but in this particular case, it seems that "core" is being imported without referencing code in it. The only reason the compiler doesn't generate the char[][] TypeInfo is because an import defines it. The compiler does work on the assumption that if a module is imported, then it will also be linked in.
>>>
>>>
>>> This core module, and the entire locale package it resides in, is /not/ imported by anything. I spelled that out clearly before. You're making an assumption it is, somehow ... well, it is not. You can deduce that from the fact that the link succeeds perfectly well without that package existing in the library.
> 
> 
> After taking a much needed break from this, I'm having another bash at it. The locale package highlighted in the last bout has been through a number of changes, and thus the behaviour will now be somewhat different than before.
> 
> For a refresher on this issue, here's an overview: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=49257 
> 
> 
> The upshot is that reordering the content as presented to the librarian now has an effect on the resultant binary size. This tells me two things:
> 
> 1) there were more than just the one compiler-generated unresolved symbol in the first case, though I did not spot it after many painstaking hours of frustrating effort. In fact, there may well have been a number of vaguely intertwined dependencies spread throughout the library, due entirely to these compiler-generated symbols.
> 
> 2) there is no feasible manner in which a developer can control how lib contents are linked while the compiler continues to generate duplicate symbols under the covers. The fact that it does this on a regular basis simply serves to highlight a critical problem.
> 
> 
>  > Then the typeinfo for char[][] is being generated by another module. I
>  > suggest it would be helpful to find that module. Grep is a handy tool
>  > for finding it.
> 
> What would be the point? These symbols are compiler generated, and exist in a variety of places because of that fact alone. Lest we forget, we're not even talking about just one symbolic name ~ the compiler generates duplicate symbols for any typeinfo that does not match the pre-packaged list (such as char[][]). The end result is a potential rats-nest of duplicate symbols, creating a maze of intricate and fragile dependencies across the lib itself.
> 
> To put things into perspective, I have absolutely no way of knowing what the real dependencies within this library actually are. As such, I'd have to describe the situation as being "out of control". This is wholly due to the duplicate compiler-generated symbols.

In fact, it is so brittle and fragile that I now cannot reproduce what's noted above. Back to square one where it does not matter if Core.obj is listed first or last in the the lib "response file" -- it always gets linked, resulting in a wildly bloated binary.

With Core.obj removed from the lib, the resultant binary is still 60KB bigger than it should be, so the problem simply moves to a different bad link-chain instead.