That's right, it's time for everyone's favourite [1] game: guess why OPTLINK's not working! [2]

*sigh*  I'm writing a math eval library.  There are two test applications.  LexerTest only touches part of the code.  AstTest touches everything.

Now, the following works and creates an executable:

dmd -ofLexerTest (appropriate .d files)

So far, so good.  I get LexerTest.map, LexerTest.obj and LexerTest.exe.
 Let's try the other one...

dmd -ofAstTest (more .d files)

This creates a legitimate-looking AstTest.obj and a completely empty AstTest.map file.

That's it.

No executable, no error message, no register dump, nothing.

Adding or removing -g, -debug, -unittest, -release, -inline, -O does nothing.  Changing the target filename does nothing useful.

There are no spurious link.exe or dmd.exe processes running.  Invoking OPTLINK directly changes nothing.

Does anyone have any idea, any idea at all, on what could be causing
this?  I've tried everything myself and several others on #d could think of.

*miserable sob*

After over five years of this sort of shit, I am so, so completely and utterly sick to death of OPTLINK.



[1] I am, of course, being sarcastic.

[2] It is, of course, entirely possible that it's not OPTLINK's fault. Frankly though, I find that hard to believe.

Daniel Keep <daniel.keep.lists@gmail.com> wrote:

> That's right, it's time for everyone's favourite [1] game: guess why
> OPTLINK's not working! [2]
[...]
> Does anyone have any idea, any idea at all, on what could be causing
> this?  I've tried everything myself and several others on #d could think of.

<channeling src="Walter">Please reduce this to a 1-line test case.
</channeling>

You know for sure OPTLINK is being run?
If so, that wisdom of Walter's is my best idea. Try and remove stuff
until it starts working, then remove other stuff until there's nothing
left to remove. I know, it's boring and should be unnecessary. Still,
sometimes it's the only thing that works.

-- 
Simen

Simen kjaeraas wrote:
> Daniel Keep <daniel.keep.lists@gmail.com> wrote:
> 
>> That's right, it's time for everyone's favourite [1] game: guess why OPTLINK's not working! [2]
> [...]
>> Does anyone have any idea, any idea at all, on what could be causing this?  I've tried everything myself and several others on #d could think of.
> 
> <channeling src="Walter">Please reduce this to a 1-line test case. </channeling>
> 
> You know for sure OPTLINK is being run?
> If so, that wisdom of Walter's is my best idea. Try and remove stuff
> until it starts working, then remove other stuff until there's nothing
> left to remove. I know, it's boring and should be unnecessary. Still,
> sometimes it's the only thing that works.
> 

Yah, it might not actually be OPTLINK for a change.

I've had problems before with DMD silently exiting without an error message and with error success and leaving behind a corrupt .obj.

Can't remember how I worked that out though.

You could try the -v switch and/or trying to dump the contents of the .obj

- --
My enormous talent is exceeded only by my outrageous laziness.
http://www.ssTk.co.uk

On 12.05.2010 13:22, Daniel Keep wrote:
>
> That's right, it's time for everyone's favourite [1] game: guess why
> OPTLINK's not working! [2]
>
> *sigh*  I'm writing a math eval library.  There are two test
> applications.  LexerTest only touches part of the code.  AstTest touches
> everything.
>
> Now, the following works and creates an executable:
>
> dmd -ofLexerTest (appropriate .d files)
>
> So far, so good.  I get LexerTest.map, LexerTest.obj and LexerTest.exe.
>   Let's try the other one...
>
> dmd -ofAstTest (more .d files)
>
> This creates a legitimate-looking AstTest.obj and a completely empty
> AstTest.map file.
>
> That's it.
>
> No executable, no error message, no register dump, nothing.
>

Post the source and I'll try to help. I like debugging weird problems. :)

div0 wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Simen kjaeraas wrote:
>> Daniel Keep <daniel.keep.lists@gmail.com> wrote:
>>
>>> That's right, it's time for everyone's favourite [1] game: guess why
>>> OPTLINK's not working! [2]
>> [...]
>>> Does anyone have any idea, any idea at all, on what could be causing
>>> this?  I've tried everything myself and several others on #d could
>>> think of.
>> <channeling src="Walter">Please reduce this to a 1-line test case.
>> </channeling>
>>
>> You know for sure OPTLINK is being run?
>> If so, that wisdom of Walter's is my best idea. Try and remove stuff
>> until it starts working, then remove other stuff until there's nothing
>> left to remove. I know, it's boring and should be unnecessary. Still,
>> sometimes it's the only thing that works.
>>
> 
> Yah, it might not actually be OPTLINK for a change.
> 
> I've had problems before with DMD silently exiting without an error
> message and with error success and leaving behind a corrupt .obj.
>

Yes, it sounds to me like a DMD stack overflow. I've seen that behaviour frequently, and optlink was never to blame. CTFE recursion is the only reported unfixed bug that does that -- but I've fixed half a dozen of them in the last year, so there may be more.

Some general replies:

There have been three responses along the lines of "are you sure OPTLINK is running?".  Quoting myself:

"Invoking OPTLINK directly changes nothing."

Or to be more specific:

> link AstTest,AstTest,,user32+kernel32/noi+tango.lib;
(zero-byte .map file pops into existence)

Yes, I'm sure.  I'm also sure it's the DigitalMars link.exe and in the correct directory.

> I've had problems before with DMD silently exiting without an error message and with error success and leaving behind a corrupt .obj.

I had a look at the .obj file (I actually used lib to stuff it into a .lib file and then dump the symbols), and it looked fine.

Well, except for this:

_D11TokenStream<80><8d><8d>5_ctorMFC6Source<f6>P<80><90><91>
  DFS8Location<80><89><89>AaYvJS6<80><85><cf>s5<f4>Zb<80>
  <99><aa>ZC<81><9a><87>

(broken across 3 lines; <xx> are bytes in hex since several of them were
non-printable.)

The same had happened to a number of symbols from tango.io.stream.Format.

In the case of the first one, the appropriate declarations (with fully-qualified module name commented) are:

> final class /*TokenStream.*/TokenStream
> {
>     alias bool function(Source, LocErr, out Token) NextToken;
>     this(Source src, NextToken next, LocErr err) { ... }
> }
>
> final class /*Source.*/Source { ... }
>
> alias void delegate(Location, char[], ...) /*Location.*/LocErr;
>
> struct /*Location.*/Location { ... }

As far as I understand D's name mangling, that symbol name is corrupt.

> CTFE recursion is the only reported unfixed bug that does that ...

There is exactly one CTFE function that calls itself.  However, it cannot recurse more than once (it does so to change types) *and* this function is used in the LexerTest program which does compile.

AstTest uses an additional CTFE function, but this just concatenates some strings together.

> Try and remove stuff until it starts working

I've gutted the changes between LexerTest and AstTest to the point that I get an .exe.  This involved stripping out more or less every global and member function that was added and every derived class.

What tipped it over into compiling was removing the reference to StructuredOutput from AstDumpVisitor.  Which is odd, because StructuredOutput is still being used in AstTest.

StructuredOutput, once gutted, is this:

> module StructuredOutput;
>
> import tango.io.model.IConduit : OutputStream;
> import tango.io.stream.Format : FormatOutput;
>
> final class StructuredOutput
> {
>     alias StructuredOutput This;
>     this(OutputStream os)
>     {
>     }
> }

I honestly can't see what it could be upset about.

> Post the source and I'll try to help. I like debugging weird problems. :)

Attached both regular and decaffeinated^Hgutted versions.

Daniel Keep wrote:
> 
> Well, except for this:
> 
> _D11TokenStream<80><8d><8d>5_ctorMFC6Source<f6>P<80><90><91>
>   DFS8Location<80><89><89>AaYvJS6<80><85><cf>s5<f4>Zb<80>
>   <99><aa>ZC<81><9a><87>
> 
> (broken across 3 lines; <xx> are bytes in hex since several of them were
> non-printable.)

> As far as I understand D's name mangling, that symbol name is corrupt.

I thought that as well, but they aren't corrupt,
according to WB they are compressed:

http://d.puremagic.com/issues/show_bug.cgi?id=2238

Strange what OPTLINK is doing; normally it crashes at the drop of a hat, so it seems unreasonable that it processes the .obj a bit then (un)successfully exits. Maybe the .obj is corrupt in some way that makes it look empty, so it does nothing.

Walter did post about converting it to C, did he finish that in the end?

- --
My enormous talent is exceeded only by my outrageous laziness.
http://www.ssTk.co.uk

Daniel Keep wrote:
> Some general replies:
> 
> There have been three responses along the lines of "are you sure OPTLINK
> is running?".  Quoting myself:
> 
> "Invoking OPTLINK directly changes nothing."
> 
> Or to be more specific:
> 
>> link AstTest,AstTest,,user32+kernel32/noi+tango.lib;
> (zero-byte .map file pops into existence)
> 
> Yes, I'm sure.  I'm also sure it's the DigitalMars link.exe and in the
> correct directory.
> 
>> I've had problems before with DMD silently exiting without an error
>> message and with error success and leaving behind a corrupt .obj.
> 
> I had a look at the .obj file (I actually used lib to stuff it into a
> .lib file and then dump the symbols), and it looked fine.
> 
> Well, except for this:
> 
> _D11TokenStream<80><8d><8d>5_ctorMFC6Source<f6>P<80><90><91>
>   DFS8Location<80><89><89>AaYvJS6<80><85><cf>s5<f4>Zb<80>
>   <99><aa>ZC<81><9a><87>
> 
> (broken across 3 lines; <xx> are bytes in hex since several of them were
> non-printable.)
> 
> The same had happened to a number of symbols from tango.io.stream.Format.
> 
> In the case of the first one, the appropriate declarations (with
> fully-qualified module name commented) are:
> 
>> final class /*TokenStream.*/TokenStream
>> {
>>     alias bool function(Source, LocErr, out Token) NextToken;
>>     this(Source src, NextToken next, LocErr err) { ... }
>> }
>>
>> final class /*Source.*/Source { ... }
>>
>> alias void delegate(Location, char[], ...) /*Location.*/LocErr;
>>
>> struct /*Location.*/Location { ... }
> 
> As far as I understand D's name mangling, that symbol name is corrupt.
> 
>> CTFE recursion is the only reported unfixed bug that does that ...
> 
> There is exactly one CTFE function that calls itself.  However, it
> cannot recurse more than once (it does so to change types) *and* this
> function is used in the LexerTest program which does compile.
> 
> AstTest uses an additional CTFE function, but this just concatenates
> some strings together.
> 
>> Try and remove stuff until it starts working
> 
> I've gutted the changes between LexerTest and AstTest to the point that
> I get an .exe.  This involved stripping out more or less every global
> and member function that was added and every derived class.
> 
> What tipped it over into compiling was removing the reference to
> StructuredOutput from AstDumpVisitor.  Which is odd, because
> StructuredOutput is still being used in AstTest.
> 
> StructuredOutput, once gutted, is this:
> 
>> module StructuredOutput;
>>
>> import tango.io.model.IConduit : OutputStream;
>> import tango.io.stream.Format : FormatOutput;
>>
>> final class StructuredOutput
>> {
>>     alias StructuredOutput This;
>>     this(OutputStream os)
>>     {
>>     }
>> }
> 
> I honestly can't see what it could be upset about.
> 
>> Post the source and I'll try to help. I like debugging weird problems.
>> :)
> 
> Attached both regular and decaffeinated^Hgutted versions.

Thanks! I hope torhu is able to create a bug report from this.

On 13.05.2010 10:39, Daniel Keep wrote:
> Attached both regular and decaffeinated^Hgutted versions.

Most likely DMD turns VisitorCtfe.d into an invalid object file. But since you don't need to link with objects that contain only ctfe functions...

Here's what I get:
------
d:\prog\test\D\matheval>dmd -c AstTest.d Ast.d AstDumpVisitor.d Lexer.d Location.d Parser.d Source.d StructuredOutput.d Tokens.d TokenStream.d VisitorCtfe.d

d:\prog\test\D\matheval>dmd -ofAstTest.exe *.obj -v
binary    d:\prog\dmd\bin\dmd.exe
version   v1.060
config    d:\prog\dmd\bin\sc.ini
d:\prog\dmd\bin\..\..\dm\bin\link.exe "*","AstTest.exe",,user32+kernel32/noi;
OPTLINK (R) for Win32  Release 8.00.1
Copyright (C) Digital Mars 1989-2004  All rights reserved.
VisitorCtfe.obj(VisitorCtfe)  Offset 004D8H Record Type 009D
 Error 16: Index Range
--- errorlevel 1

d:\prog\test\D\matheval>del VisitorCtfe.obj

d:\prog\test\D\matheval>dmd -ofAstTest.exe *.obj -v
binary    d:\prog\dmd\bin\dmd.exe
version   v1.060
config    d:\prog\dmd\bin\sc.ini
d:\prog\dmd\bin\..\..\dm\bin\link.exe "*","AstTest.exe",,user32+kernel32/noi;

d:\prog\test\D\matheval>AstTest.exe
Can't initialize the TangoTrace LGPL stuff
Usage: AstTest.exe CODE

d:\prog\test\D\matheval>YAY
--------

Seems I've got an older version of optlink, 8.00.1.  With 8.00.2 I get the same behavior you did, but it still works if you don't link with VisitorCtfe.obj.

When running 8.00.2 in the msvc debugger, it just looks like optlink successfully exits.  No crash or anything.  But no useful output either.

So I guess this is two bug reports, not just one...