"Walter Bright" <newshound2@digitalmars.com> wrote in message news:khaplp$nn5$1@digitalmars.com...
> On 3/7/2013 4:35 AM, Daniel Murphy wrote:
>> Maybe I'm missing something, but I don't understand why this is
>> necessary.
>> It should be trivial to look at the map file for the closest preceding
>> label.  On modern toolchains it should even be possible to compile the
>> asm
>> with debug line information.  Or is grepping more effective than it
>> sounds?
>
> Modern operating systems load programs at different base addresses each time, so the addresses in the map file don't match up with the addresses at run time.
>

You can turn windows 7's aslr off.

> The older assembler I was using didn't emit symbolic debug info. I haven't tried out ML that way.
>

Looks like the /Zd swich gives you line numbers.

>
>> This sounds completely insane. (But also fun, in an extremely frustrating
>> way)
>> I suspect it would be easier to reverse-engineer microsoft's (or
>> borland's?)
>> old linker from the disassembly, if they wrote it in C.
>
> You'd be incorrect if you thought MS's or Borland's linker was better. Optlink sold very, very well against those linkers back in the day, because Optlink was far better.
>

That's not what I meant.  I mean that when trying to work out what the assembly does, having real stack frames and a consistent calling convention could be more useful than knowing the label names and having out of date comments.

Optlink is lightning fast and memory efficient, but honestly most of the time you don't care.  Computers these days are ridiculously overpowered for something as simple as linking.

In huge projects this matters more, but then you start running into optlink's limitations.

>
> There's much more to making a linker. There's the debug info (which you mentioned), then there are the resource files, the module definition files, and lastly the myriad of rather complex switches that are expected. Almost none of that is tested by the dmd test suite.
>

Yeah, debug information is fairly nasty.  It is at least as hard to implement as the actual linking.  (I have done some work on this for CV4, just not finished)

I don't expect res and def support are anywhere near that complicated.  Same goes for most of the command line switches.  (I have read through the list, but it's possible I'm being overconfident here)

> However, Optlink does bring considerable value to any linker replacement project, in that embedded in it is an enormous amount of lore about all those twinkie little things that matter, and how things really need to work.

Optlink also has a lot of support for dead file formats/memory models and an insanely convoluted code base.  Like you said, porting to C doesn't escape this.

On 3/7/2013 2:25 PM, Daniel Murphy wrote:
> That's not what I meant.  I mean that when trying to work out what the
> assembly does, having real stack frames and a consistent calling convention
> could be more useful than knowing the label names and having out of date
> comments.

I've done some disassembly of real programs. It is a LOT harder than dealing with Optlink. Really, it's terribly difficult. Having the symbolic names, a logical organization, and even out of date comments, are a huge help.


> I don't expect res and def support are anywhere near that complicated.  Same
> goes for most of the command line switches.  (I have read through the list,
> but it's possible I'm being overconfident here)

The major problem is the lack of a test suite for those things.


>> However, Optlink does bring considerable value to any linker replacement
>> project, in that embedded in it is an enormous amount of lore about all
>> those twinkie little things that matter, and how things really need to
>> work.
>
> Optlink also has a lot of support for dead file formats/memory models and an
> insanely convoluted code base.  Like you said, porting to C doesn't escape
> this.

In my work converting Optlink to C, I dropped support for the false compile time conditionals. There's no rationale for building a real mode optlink executable. There's also no reason to support building OS/2 executables anymore, etc.

On 3/7/13, Walter Bright <newshound2@digitalmars.com> wrote:
> 1. all the supported file formats were completely encapsulated and
> modularized
> For some idea of how that can work, see libmscoff.c and scanmscoff.c

DDL seems to do a decent job of abstracting things, e.g.:

OMF loader: http://dsource.org/projects/ddl/browser/trunk/ddl/omf COFF loader: http://dsource.org/projects/ddl/browser/trunk/ddl/coff And the root dir: http://dsource.org/projects/ddl/browser/trunk/ddl

Anyway I'm not suggesting it's a complete implementation, but it might be worth taking a look. It's BSD licensed, hopefully that means it's ok to look at and maybe extract some useful code or ideas from it.
>From the source comments it appears Don Clugston also worked on this
for some time.

"Walter Bright" <newshound2@digitalmars.com> wrote in message news:khb5d7$1cfd$1@digitalmars.com...
> On 3/7/2013 2:25 PM, Daniel Murphy wrote:
>> That's not what I meant.  I mean that when trying to work out what the
>> assembly does, having real stack frames and a consistent calling
>> convention
>> could be more useful than knowing the label names and having out of date
>> comments.
>
> I've done some disassembly of real programs. It is a LOT harder than dealing with Optlink. Really, it's terribly difficult. Having the symbolic names, a logical organization, and even out of date comments, are a huge help.
>

I've never worked on assembly that didn't use standard calling conventions, so I don't know how difficult it is.

>
>> I don't expect res and def support are anywhere near that complicated.
>> Same
>> goes for most of the command line switches.  (I have read through the
>> list,
>> but it's possible I'm being overconfident here)
>
> The major problem is the lack of a test suite for those things.
>

Makes sense.  I have no idea how big the dmd/dmc/msc subset that would need to be supported is.

>
>>> However, Optlink does bring considerable value to any linker replacement project, in that embedded in it is an enormous amount of lore about all those twinkie little things that matter, and how things really need to work.
>>
>> Optlink also has a lot of support for dead file formats/memory models and
>> an
>> insanely convoluted code base.  Like you said, porting to C doesn't
>> escape
>> this.
>
> In my work converting Optlink to C, I dropped support for the false compile time conditionals. There's no rationale for building a real mode optlink executable. There's also no reason to support building OS/2 executables anymore, etc.
>

Good, but does the code still increase the difficulty in porting?

And even once it's in C, optlink will probably never be more than a win32/omf linker.

On 3/7/2013 7:09 PM, Daniel Murphy wrote:
> Good, but does the code still increase the difficulty in porting?

I don't understand your question.


> And even once it's in C, optlink will probably never be more than a
> win32/omf linker.

That's correct. However, it'll be much more maintainable, and it'll be a gold mine of information about linker trivia on how to do things with obscure/undocumented/bizarre file formats.

"Walter Bright" <newshound2@digitalmars.com> wrote in message news:khblbe$27f5$1@digitalmars.com...
> On 3/7/2013 7:09 PM, Daniel Murphy wrote:
>> Good, but does the code still increase the difficulty in porting?
>
> I don't understand your question.
>

Does the presence of support for eg. linking OS2 executables make the codebase harder to understand?

>
>> And even once it's in C, optlink will probably never be more than a win32/omf linker.
>
> That's correct. However, it'll be much more maintainable,

I don't know how much redesign you're planning, but I can't imagine it ever being as maintainable as a pure d codebase.  A less stable/complete linker that attracts more contributors should overtake a more stable linker with only a couple of developers that grok it.

> and it'll be a gold mine of information about linker trivia on how to do things with obscure/undocumented/bizarre file formats.
>

What is the license on optlink?  Can other linkers actually use this information?

On 3/7/2013 7:27 PM, Daniel Murphy wrote:
> "Walter Bright" <newshound2@digitalmars.com> wrote in message
> news:khblbe$27f5$1@digitalmars.com...
>> On 3/7/2013 7:09 PM, Daniel Murphy wrote:
>>> Good, but does the code still increase the difficulty in porting?
>>
>> I don't understand your question.
>>
>
> Does the presence of support for eg. linking OS2 executables make the
> codebase harder to understand?

Yes.


>> That's correct. However, it'll be much more maintainable,
>
> I don't know how much redesign you're planning, but I can't imagine it ever
> being as maintainable as a pure d codebase.  A less stable/complete linker
> that attracts more contributors should overtake a more stable linker with
> only a couple of developers that grok it.

That's true, but we don't have that other linker yet.

The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost benefit moving forward? I think it's easier to just fix optlink's bugs.

I don't want to discourage people from trying to come up with a replacement linker for win32 written in D. I think that is a great project. But while a linker is a conceptually simple program, the awful file formats involved make it unnecessarily difficult and there are simply a lot of details and other things one has to do.

Like I said before, it'll take a sustained and determined effort to come up with a viable replacement for optlink.


> What is the license on optlink?

Same as the dmd back end.


> Can other linkers actually use this information?

They can use the information, yes, but not the code.

"Walter Bright" <newshound2@digitalmars.com> wrote in message news:khbmkn$298s$1@digitalmars.com...
>
>>> That's correct. However, it'll be much more maintainable,
>>
>> I don't know how much redesign you're planning, but I can't imagine it
>> ever
>> being as maintainable as a pure d codebase.  A less stable/complete
>> linker
>> that attracts more contributors should overtake a more stable linker with
>> only a couple of developers that grok it.
>
> That's true, but we don't have that other linker yet.
>
> The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost benefit moving forward? I think it's easier to just fix optlink's bugs.
>

You're probably right, but it would still be awesome to have our own modern linker.  Just being able to use coff import libraries directly on win32 would be very nice.

> I don't want to discourage people from trying to come up with a replacement linker for win32 written in D. I think that is a great project. But while a linker is a conceptually simple program, the awful file formats involved make it unnecessarily difficult and there are simply a lot of details and other things one has to do.
>
> Like I said before, it'll take a sustained and determined effort to come up with a viable replacement for optlink.
>

Agreed.

>
>> What is the license on optlink?
>
> Same as the dmd back end.
>
>
>> Can other linkers actually use this information?
>
> They can use the information, yes, but not the code.
>

That's interesting, I didn't realise this.

On 3/7/2013 7:41 PM, Walter Bright wrote:
> On 3/7/2013 7:27 PM, Daniel Murphy wrote:
>> "Walter Bright" <newshound2@digitalmars.com> wrote in message news:khblbe$27f5$1@digitalmars.com...
>>> On 3/7/2013 7:09 PM, Daniel Murphy wrote:
>>> That's correct. However, it'll be much more maintainable,
>>
>> I don't know how much redesign you're planning, but I can't imagine it ever being as maintainable as a pure d codebase.  A less stable/complete linker that attracts more contributors should overtake a more stable linker with only a couple of developers that grok it.
> 
> That's true, but we don't have that other linker yet.
> 
> The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost benefit moving forward? I think it's easier to just fix optlink's bugs.
> 
> I don't want to discourage people from trying to come up with a replacement linker for win32 written in D. I think that is a great project. But while a linker is a conceptually simple program, the awful file formats involved make it unnecessarily difficult and there are simply a lot of details and other things one has to do.
> 
> Like I said before, it'll take a sustained and determined effort to come up with a viable replacement for optlink.

Personally, even though I don't use win32, I believe that moving it over to use the VS toolchain and runtime is the right path forward.

On Friday, 8 March 2013 at 04:17:03 UTC, Brad Roberts wrote:
> On 3/7/2013 7:41 PM, Walter Bright wrote:
>> On 3/7/2013 7:27 PM, Daniel Murphy wrote:
>>> "Walter Bright" <newshound2@digitalmars.com> wrote in message
>>> news:khblbe$27f5$1@digitalmars.com...
>>>> On 3/7/2013 7:09 PM, Daniel Murphy wrote:
>>>> That's correct. However, it'll be much more maintainable,
>>>
>>> I don't know how much redesign you're planning, but I can't imagine it ever
>>> being as maintainable as a pure d codebase.  A less stable/complete linker
>>> that attracts more contributors should overtake a more stable linker with
>>> only a couple of developers that grok it.
>> 
>> That's true, but we don't have that other linker yet.
>> 
>> The other thing is, we just don't have a need for our own linker for any platform other than win32. So what's the cost
>> benefit moving forward? I think it's easier to just fix optlink's bugs.
>> 
>> I don't want to discourage people from trying to come up with a replacement linker for win32 written in D. I think that
>> is a great project. But while a linker is a conceptually simple program, the awful file formats involved make it
>> unnecessarily difficult and there are simply a lot of details and other things one has to do.
>> 
>> Like I said before, it'll take a sustained and determined effort to come up with a viable replacement for optlink.
>
> Personally, even though I don't use win32, I believe that moving it over to use the VS toolchain and runtime is the
> right path forward.

I've actually modified dmd's frontend to support using the VS tool chain for linking 32 bit executables. But I didn't understand the backend to seperate the 32bit binary generation vs 64bit. I believe it shouldn't be too much. But I really don't understand it so if anyone is willing to help on this.
I haven't broken 64bit generation as far as I know. Because I added a cli argument (-msvc) which is enabled for -m64 automatically on Windows.