Some months ago, I did make the source to optlink available on github:

https://github.com/DigitalMars/optlink

Rainer Schuetze has improved it where it can be built with modern tools (the older tools would not run on Win7). I know some of you are frustrated by optlink problems. Well, now you can do something about it!

Note that the optlink source is quite a challenge to work on. It's very old skool (80's style).

Here's an approach that I've used successfully to fix optlink seg faults:

1. Find out where in the source code it is. This is not so easy, even if using a debugger. The trouble is the asm functions do not have standard stack frames, so the debugger cannot do a stack trace. Even where there is a standard stack frame, the modern Microsoft debuggers fail to recognize them. You gotta use an older debugger under Windows XP.

So, what I do, is look at the asm code where the seg fault occurs, and then grep through the asm source till I find it.

2. Convert the source file where the seg fault occurs to C. I do this one function at a time, running the full set of tests at each step. Otherwise, it's just impossible to figure out where you made a mistake in the conversion. Sometimes, you gotta go even finer grained - using the inline assembler feature of dmc to convert asm code line by line.

You gotta pay very, very close attention to which registers have parameters passed through them, which registers are saved, and which registers have parameters that are silently passed through a function to callees of that function (!). This information is all pretty much utterly lacking in the comments, and the comments are often dead wrong as they refer to much older versions of the code.

3. Once it is in C, things get a lot easier. You can, for example, insert printf's to figure out where things go wrong, and *then* fix it.


The full test suite for optlink has a lot of stuff I cannot publish on github. However, what you can do is run the win32 tests for dmd. I, of course, will run the full one when verifying pulls.

Happy hacking!

Walter Bright:

> Happy hacking!

Extra karma points if done blindfolded, using a Braille tablet :-)

Bye,
bearophile

On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
> Some months ago, I did make the source to optlink available on github:
>
> https://github.com/DigitalMars/optlink
>
> Rainer Schuetze has improved it where it can be built with modern tools (the older tools would not run on Win7). I know some of you are frustrated by optlink problems. Well, now you can do something about it!
>
> Note that the optlink source is quite a challenge to work on. It's very old skool (80's style).
>
> Here's an approach that I've used successfully to fix optlink seg faults:
>
> 1. Find out where in the source code it is. This is not so easy, even if using a debugger. The trouble is the asm functions do not have standard stack frames, so the debugger cannot do a stack trace. Even where there is a standard stack frame, the modern Microsoft debuggers fail to recognize them. You gotta use an older debugger under Windows XP.
>
> So, what I do, is look at the asm code where the seg fault occurs, and then grep through the asm source till I find it.
>
> 2. Convert the source file where the seg fault occurs to C. I do this one function at a time, running the full set of tests at each step. Otherwise, it's just impossible to figure out where you made a mistake in the conversion. Sometimes, you gotta go even finer grained - using the inline assembler feature of dmc to convert asm code line by line.
>
> You gotta pay very, very close attention to which registers have parameters passed through them, which registers are saved, and which registers have parameters that are silently passed through a function to callees of that function (!). This information is all pretty much utterly lacking in the comments, and the comments are often dead wrong as they refer to much older versions of the code.
>
> 3. Once it is in C, things get a lot easier. You can, for example, insert printf's to figure out where things go wrong, and *then* fix it.
>
>
> The full test suite for optlink has a lot of stuff I cannot publish on github. However, what you can do is run the win32 tests for dmd. I, of course, will run the full one when verifying pulls.
>
> Happy hacking!

It sound like a nightmare. What is the reason in the first place to use optlink ? Isn't it possible to use another linker ?

On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
> Some months ago, I did make the source to optlink available on github:
>
> [...]
>
> Happy hacking!

This reminds me of http://www.catb.org/jargon/html/story-of-mel.html

:)

Lars

On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad wrote:
> On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
> >Some months ago, I did make the source to optlink available on github:
> >
> >[...]
> >
> >Happy hacking!
> 
> This reminds me of http://www.catb.org/jargon/html/story-of-mel.html
[...]

Man, that brings back the memories. I remember doing those kinds of stunts when coding for the good ole Motorola 6502. Back in those days, Apple actually published the full listing of their ROM code along with the instruction specs, including the number of cycles per instruction, etc.. There were many a fun night plotting out the optimal instruction sequences and spacing out instructions just right so the exact timing is achieved. Lots of fun.


T

-- 
What doesn't kill me makes me stranger.

On Thu, 07 Mar 2013 06:04:29 +0100, deadalnix <deadalnix@gmail.com> wrote:

> It sound like a nightmare. What is the reason in the first place to use optlink ? Isn't it possible to use another linker ?

It's fast, it's pretty well tested, Walter's written it himself, and
changing to a different linker is likely to have its own set of challenges.

-- 
Simen

On Thursday, 7 March 2013 at 01:25:02 UTC, Walter Bright wrote:
>
> https://github.com/DigitalMars/optlink
>
> Rainer Schuetze has improved it where it can be built with modern tools (the older tools would not run on Win7). I know some of you are frustrated by optlink problems. Well, now you can do something about it!

Sorry if this has been answered before/is common knowledge, but is porting functions at a time to C wanted for optlink in general, or only
for finding segfaults? (e.g. are pull-requests for that welcome)

On 2013-03-07 11:12, Moritz Maxeiner wrote:

> Sorry if this has been answered before/is common knowledge, but is
> porting functions at a time to C wanted for optlink in general, or only
> for finding segfaults? (e.g. are pull-requests for that welcome)

Yes, in general. I think the idea is to port the whole Optlink to C and then to D. It's easier to port from C to D than from assembly to D. This is because you can use a C version that doesn't use the runtime or standard library to get the generate assembly as close as possible to the original one. I think it was something like that.

-- 
/Jacob Carlborg

"Walter Bright" <newshound2@digitalmars.com> wrote in message news:kh8q9e$2j02$1@digitalmars.com...
> Some months ago, I did make the source to optlink available on github:
>
> https://github.com/DigitalMars/optlink
>
[snip]
>
> So, what I do, is look at the asm code where the seg fault occurs, and then grep through the asm source till I find it.
>

Maybe I'm missing something, but I don't understand why this is necessary. It should be trivial to look at the map file for the closest preceding label.  On modern toolchains it should even be possible to compile the asm with debug line information.  Or is grepping more effective than it sounds?

> 2. Convert the source file where the seg fault occurs to C. I do this one function at a time, running the full set of tests at each step. Otherwise, it's just impossible to figure out where you made a mistake in the conversion. Sometimes, you gotta go even finer grained - using the inline assembler feature of dmc to convert asm code line by line.
>
> You gotta pay very, very close attention to which registers have parameters passed through them, which registers are saved, and which registers have parameters that are silently passed through a function to callees of that function (!). This information is all pretty much utterly lacking in the comments, and the comments are often dead wrong as they refer to much older versions of the code.
>

This sounds completely insane. (But also fun, in an extremely frustrating
way)
I suspect it would be easier to reverse-engineer microsoft's (or borland's?)
old linker from the disassembly, if they wrote it in C.

> 3. Once it is in C, things get a lot easier. You can, for example, insert printf's to figure out where things go wrong, and *then* fix it.
>
>
> The full test suite for optlink has a lot of stuff I cannot publish on github. However, what you can do is run the win32 tests for dmd. I, of course, will run the full one when verifying pulls.
>
> Happy hacking!

As a side note I actually wrote an omf (dmd-emitted subset) linker last year that gets through almost all of the dmd test suite.  It has no debug info support, which sucks, but it may work in some cases where optlink can't cope.  It is fairly naive performance-wise, but is written in a much friendlier language.

If anyone is interested I'll put it up on github.

On Thu, Mar 07, 2013 at 07:09:22AM +0100, Lars T. Kyllingstad wrote:

> This reminds me of http://www.catb.org/jargon/html/story-of-mel.html

Good story! Imagine how Mel must have known the entire program
by heart and he never got any change requests that are so common today.
I would have been addicted to that, too. But luckily 1975 was 8
years before my birth and so I started "hacking" with QBasic
Gorillaz, a turn based strategy game.
I guess with SSE it still today makes sense to count
instruction delays and throughput. Also take a look at this:
http://research.scee.net/files/presentations/gcapaustralia09/Pitfalls_of_Object_Oriented_Programming_GCAP_09.pdf
I don't think people like this programmer Mel are job-less
today.

-- 
Marco