This is a lengthy one, for which I apologize. I've tried to lay the problem out as best I can, but I am not sure exactly where the problem lies. I am reasonably certain that it is a compilation or linkage problem, and not a problem with the code.

I have done some work with CeleriD and added support for GDC. This is required for Linux support, as DMD can't build shared objects on Linux. However, I've run into some basic building issues that I could use some help with.

Before I started mucking around with it, CeleriD simply took all of the code files needed, jammed them all on a command line with the appropriate options (-c, mostly), and compiled them. Then it linked them together. Fine.

When adding gdc support, the first thing I realised was that gdc (and gcc) doesn't support DMD's -od option, and distutils requires that all of the object files be placed in a specific directory. My first stab at getting around this was to compile the sources one at a time, specifying the location of the object files one by one.

When I tested this with DMD, it compiled but did not link. It complained of undefined symbols. The symbols given were the big long mangles of template instances. I am still not sure why this happened. I fixed it by putting it back the way it was, as a single call to the compiler.

To get rid of the need for the -od option, I changed the current working directory to where the object files needed to go, and specified the full path to each of the source files. By not specifying a location, the object files just get dumped in the cwd. Problem solved.

Except that gdc still doesn't work. The .so compiles and links nicely enough, except that when it is imported by Python, Python issues this error:

ImportError: /full/path/to/testdll.so: undefined symbol: _D3pyd9func_wrap56__T9func_wrapS25_D7testdll7dg_testFDFZvZvVk1TvTPFDFZvZvZ4funcUPS6python8PyObjectPS6python8PyObjectZPS6python8PyObject

Which is the mangle of a template instance. One of the very ones, I might add, that Optlink wasn't finding when I compiled the source files one by one.

For reference, the calls to the compiler and linker are:

DMD in Windows:
compiler:
>C:\dmd\dmd\bin\dmd.exe
-c
-debug -unittest
-I<location of the Python header>
-I<location of Pyd>
<all of the source files: Python, Pyd, boilerplate, user code>

linker:
>C:\dmd\dmd\bin\dmd.exe
-debug -unittest
-ofbuild\lib.win32-2.4\testdll.pyd
<the python DLL's .lib file>
<a list of the .obj files>
<the generated .def file>

That /works/. This doesn't:

GDC in Linux:
compiler:
$ gdc
-fPIC -c
-fdebug -funittest
-I <location of the Python header>
-I <location of Pyd>
<all of the source files: Python, Pyd, boilerplate, user code>

linker:
$ gdc
-fPIC -nostartfiles -shared
-fdebug -funittest
-o build/lib.linux-i686-2.4/testdll.so
<a list of the .o files>
-Wl,-soname,testdll.so
-lpython2.4

-nostartfiles is needed to override the _init and _fini functions, which are called when the module is loaded and unloaded (they are defined in the boilerplate code). -shared and the -soname option are similar to the .def file in Windows. (They tell the linker to make a .so file.) The final option is like the .lib file in Windows; it links it against Python.

It is possible that this is a problem with gdc? Or am I making some stupid mistake?

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki

Kirk McDonald wrote:

> This is a lengthy one, for which I apologize. I've tried to lay the problem out as best I can, but I am not sure exactly where the problem lies. I am reasonably certain that it is a compilation or linkage problem, and not a problem with the code.
<snip>
> It is possible that this is a problem with gdc? Or am I making some stupid mistake?
> 

No, you don't make any stupid mistakes. I call this a rather bad bug in the DMD frontend (which is why it is possible to see it with both compilers). The problem is that the template instances usually don't belong in the object/source file of the template itself, but in the client code. Thus the same instance (that is, the template is instantiated with the same type/parameters), might be needed in several object files. AFAIK, this is not handled (or definately not handled correctly) when compiling one by one object file, and which is why you won't see any D libs (that is, foo.lib, or libfoo.a files) with templates around. However, DMD is able to process all the input .d files at once and in this mode it obviously operate differently, and when producing the executable, it know everything it needs to know about the template instance, and thus it is no longer a problem.

So, there a couple of non-perfect options, and the least perfect is not generating the symbols at all :P The other is duplicating them over the object files that needs the instance, but then the linker usually don't know which one to choose. Similar problems can be found with the moduleinfo symbols in some cases and when used with libraries. The problems seems to be worse on linux, probably because Walter knows better how to use the OMF format to the fullest compared to the ELF format.

There should be quite a few bug reports around concerning both of these issues.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi

Lars Ivar Igesund wrote:
> Kirk McDonald wrote:
> 
> 
>>This is a lengthy one, for which I apologize. I've tried to lay the
>>problem out as best I can, but I am not sure exactly where the problem
>>lies. I am reasonably certain that it is a compilation or linkage
>>problem, and not a problem with the code.
> 
> <snip>
> 
>>It is possible that this is a problem with gdc? Or am I making some
>>stupid mistake?
>>
> 
> 
> No, you don't make any stupid mistakes. I call this a rather bad bug in the
> DMD frontend (which is why it is possible to see it with both compilers). The problem is that the template instances usually don't belong in the
> object/source file of the template itself, but in the client code. Thus the
> same instance (that is, the template is instantiated with the same
> type/parameters), might be needed in several object files. AFAIK, this is
> not handled (or definately not handled correctly) when compiling one by one
> object file, and which is why you won't see any D libs (that is, foo.lib,
> or libfoo.a files) with templates around. However, DMD is able to process
> all the input .d files at once and in this mode it obviously operate
> differently, and when producing the executable, it know everything it needs
> to know about the template instance, and thus it is no longer a problem.

Curses. I was afraid it was something like that. And it seems plain that GDC doesn't operate in this mode, for whatever reason, and so the problem doesn't go away when I specify the source files all at once.

> 
> So, there a couple of non-perfect options, and the least perfect is not
> generating the symbols at all :P The other is duplicating them over the
> object files that needs the instance, but then the linker usually don't
> know which one to choose. Similar problems can be found with the moduleinfo
> symbols in some cases and when used with libraries. The problems seems to
> be worse on linux, probably because Walter knows better how to use the OMF
> format to the fullest compared to the ELF format.
> 
> There should be quite a few bug reports around concerning both of these
> issues.
> 

I suppose now I will try to find them, and pester Walter about them.

Well, Pyd should be ready with Linux support whenever a compiler is available that resolves this problem. The code is now in place for it.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki

Kirk McDonald wrote:
> Curses. I was afraid it was something like that. And it seems plain that GDC doesn't operate in this mode, for whatever reason, and so the problem doesn't go away when I specify the source files all at once.

IIRC, David had once said gdc operates upon only one module at a time

kris wrote:

> Kirk McDonald wrote:
>> Curses. I was afraid it was something like that. And it seems plain that GDC doesn't operate in this mode, for whatever reason, and so the problem doesn't go away when I specify the source files all at once.
> 
> IIRC, David had once said gdc operates upon only one module at a time

And the reason for this, is gcc which operates upon only one module at the time, IIRC.
-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi

Kirk McDonald escribió:
> 
> Before I started mucking around with it, CeleriD simply took all of the code files needed, jammed them all on a command line with the appropriate options (-c, mostly), and compiled them. Then it linked them together. Fine.
> 
> When adding gdc support, the first thing I realised was that gdc (and gcc) doesn't support DMD's -od option, and distutils requires that all of the object files be placed in a specific directory. My first stab at getting around this was to compile the sources one at a time, specifying the location of the object files one by one.
> 
> When I tested this with DMD, it compiled but did not link. It complained of undefined symbols. The symbols given were the big long mangles of template instances. I am still not sure why this happened. I fixed it by putting it back the way it was, as a single call to the compiler.
> 
> To get rid of the need for the -od option, I changed the current working directory to where the object files needed to go, and specified the full path to each of the source files. By not specifying a location, the object files just get dumped in the cwd. Problem solved.
> 

Not too helpful, but anyway: if you call gdc as "gdmd", it accepts dmd's parameters (instead of gcc's). gdmd is just a wrapper script.

-- 
Carlos Santander Bernal

Carlos Santander wrote:
> Kirk McDonald escribió:
> 
>>
>> Before I started mucking around with it, CeleriD simply took all of the code files needed, jammed them all on a command line with the appropriate options (-c, mostly), and compiled them. Then it linked them together. Fine.
>>
>> When adding gdc support, the first thing I realised was that gdc (and gcc) doesn't support DMD's -od option, and distutils requires that all of the object files be placed in a specific directory. My first stab at getting around this was to compile the sources one at a time, specifying the location of the object files one by one.
>>
>> When I tested this with DMD, it compiled but did not link. It complained of undefined symbols. The symbols given were the big long mangles of template instances. I am still not sure why this happened. I fixed it by putting it back the way it was, as a single call to the compiler.
>>
>> To get rid of the need for the -od option, I changed the current working directory to where the object files needed to go, and specified the full path to each of the source files. By not specifying a location, the object files just get dumped in the cwd. Problem solved.
>>
> 
> Not too helpful, but anyway: if you call gdc as "gdmd", it accepts dmd's parameters (instead of gcc's). gdmd is just a wrapper script.
> 

I'm well aware of gdmd, but I have chosen to not use it. It is a perl script, and will not typically work on Windows. I eventually want to support using GDC on Windows.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki