View mode: basic / threaded / horizontal-split · Log in · Help
August 08, 2006
Pyd on Linux building woes
This is a lengthy one, for which I apologize. I've tried to lay the 
problem out as best I can, but I am not sure exactly where the problem 
lies. I am reasonably certain that it is a compilation or linkage 
problem, and not a problem with the code.

I have done some work with CeleriD and added support for GDC. This is 
required for Linux support, as DMD can't build shared objects on Linux. 
However, I've run into some basic building issues that I could use some 
help with.

Before I started mucking around with it, CeleriD simply took all of the 
code files needed, jammed them all on a command line with the 
appropriate options (-c, mostly), and compiled them. Then it linked them 
together. Fine.

When adding gdc support, the first thing I realised was that gdc (and 
gcc) doesn't support DMD's -od option, and distutils requires that all 
of the object files be placed in a specific directory. My first stab at 
getting around this was to compile the sources one at a time, specifying 
the location of the object files one by one.

When I tested this with DMD, it compiled but did not link. It complained 
of undefined symbols. The symbols given were the big long mangles of 
template instances. I am still not sure why this happened. I fixed it by 
putting it back the way it was, as a single call to the compiler.

To get rid of the need for the -od option, I changed the current working 
directory to where the object files needed to go, and specified the full 
path to each of the source files. By not specifying a location, the 
object files just get dumped in the cwd. Problem solved.

Except that gdc still doesn't work. The .so compiles and links nicely 
enough, except that when it is imported by Python, Python issues this error:

ImportError: /full/path/to/testdll.so: undefined symbol: 
_D3pyd9func_wrap56__T9func_wrapS25_D7testdll7dg_testFDFZvZvVk1TvTPFDFZvZvZ4funcUPS6python8PyObjectPS6python8PyObjectZPS6python8PyObject

Which is the mangle of a template instance. One of the very ones, I 
might add, that Optlink wasn't finding when I compiled the source files 
one by one.

For reference, the calls to the compiler and linker are:

DMD in Windows:
compiler:
>C:\dmd\dmd\bin\dmd.exe
-c
-debug -unittest
-I<location of the Python header>
-I<location of Pyd>
<all of the source files: Python, Pyd, boilerplate, user code>

linker:
>C:\dmd\dmd\bin\dmd.exe
-debug -unittest
-ofbuild\lib.win32-2.4\testdll.pyd
<the python DLL's .lib file>
<a list of the .obj files>
<the generated .def file>

That /works/. This doesn't:

GDC in Linux:
compiler:
$ gdc
-fPIC -c
-fdebug -funittest
-I <location of the Python header>
-I <location of Pyd>
<all of the source files: Python, Pyd, boilerplate, user code>

linker:
$ gdc
-fPIC -nostartfiles -shared
-fdebug -funittest
-o build/lib.linux-i686-2.4/testdll.so
<a list of the .o files>
-Wl,-soname,testdll.so
-lpython2.4

-nostartfiles is needed to override the _init and _fini functions, which 
are called when the module is loaded and unloaded (they are defined in 
the boilerplate code). -shared and the -soname option are similar to the 
.def file in Windows. (They tell the linker to make a .so file.) The 
final option is like the .lib file in Windows; it links it against Python.

It is possible that this is a problem with gdc? Or am I making some 
stupid mistake?

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki
August 08, 2006
Re: Pyd on Linux building woes
Kirk McDonald wrote:

> This is a lengthy one, for which I apologize. I've tried to lay the
> problem out as best I can, but I am not sure exactly where the problem
> lies. I am reasonably certain that it is a compilation or linkage
> problem, and not a problem with the code.
<snip>
> It is possible that this is a problem with gdc? Or am I making some
> stupid mistake?
> 

No, you don't make any stupid mistakes. I call this a rather bad bug in the
DMD frontend (which is why it is possible to see it with both compilers). 
The problem is that the template instances usually don't belong in the
object/source file of the template itself, but in the client code. Thus the
same instance (that is, the template is instantiated with the same
type/parameters), might be needed in several object files. AFAIK, this is
not handled (or definately not handled correctly) when compiling one by one
object file, and which is why you won't see any D libs (that is, foo.lib,
or libfoo.a files) with templates around. However, DMD is able to process
all the input .d files at once and in this mode it obviously operate
differently, and when producing the executable, it know everything it needs
to know about the template instance, and thus it is no longer a problem.

So, there a couple of non-perfect options, and the least perfect is not
generating the symbols at all :P The other is duplicating them over the
object files that needs the instance, but then the linker usually don't
know which one to choose. Similar problems can be found with the moduleinfo
symbols in some cases and when used with libraries. The problems seems to
be worse on linux, probably because Walter knows better how to use the OMF
format to the fullest compared to the ELF format.

There should be quite a few bug reports around concerning both of these
issues.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi
August 08, 2006
Re: Pyd on Linux building woes
Lars Ivar Igesund wrote:
> Kirk McDonald wrote:
> 
> 
>>This is a lengthy one, for which I apologize. I've tried to lay the
>>problem out as best I can, but I am not sure exactly where the problem
>>lies. I am reasonably certain that it is a compilation or linkage
>>problem, and not a problem with the code.
> 
> <snip>
> 
>>It is possible that this is a problem with gdc? Or am I making some
>>stupid mistake?
>>
> 
> 
> No, you don't make any stupid mistakes. I call this a rather bad bug in the
> DMD frontend (which is why it is possible to see it with both compilers). 
> The problem is that the template instances usually don't belong in the
> object/source file of the template itself, but in the client code. Thus the
> same instance (that is, the template is instantiated with the same
> type/parameters), might be needed in several object files. AFAIK, this is
> not handled (or definately not handled correctly) when compiling one by one
> object file, and which is why you won't see any D libs (that is, foo.lib,
> or libfoo.a files) with templates around. However, DMD is able to process
> all the input .d files at once and in this mode it obviously operate
> differently, and when producing the executable, it know everything it needs
> to know about the template instance, and thus it is no longer a problem.

Curses. I was afraid it was something like that. And it seems plain that 
GDC doesn't operate in this mode, for whatever reason, and so the 
problem doesn't go away when I specify the source files all at once.

> 
> So, there a couple of non-perfect options, and the least perfect is not
> generating the symbols at all :P The other is duplicating them over the
> object files that needs the instance, but then the linker usually don't
> know which one to choose. Similar problems can be found with the moduleinfo
> symbols in some cases and when used with libraries. The problems seems to
> be worse on linux, probably because Walter knows better how to use the OMF
> format to the fullest compared to the ELF format.
> 
> There should be quite a few bug reports around concerning both of these
> issues.
> 

I suppose now I will try to find them, and pester Walter about them.

Well, Pyd should be ready with Linux support whenever a compiler is 
available that resolves this problem. The code is now in place for it.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki
August 08, 2006
Re: Pyd on Linux building woes
Kirk McDonald wrote:
> Curses. I was afraid it was something like that. And it seems plain that 
> GDC doesn't operate in this mode, for whatever reason, and so the 
> problem doesn't go away when I specify the source files all at once.

IIRC, David had once said gdc operates upon only one module at a time
August 08, 2006
Re: Pyd on Linux building woes
kris wrote:

> Kirk McDonald wrote:
>> Curses. I was afraid it was something like that. And it seems plain that
>> GDC doesn't operate in this mode, for whatever reason, and so the
>> problem doesn't go away when I specify the source files all at once.
> 
> IIRC, David had once said gdc operates upon only one module at a time

And the reason for this, is gcc which operates upon only one module at the
time, IIRC.
-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi
August 09, 2006
Re: Pyd on Linux building woes
Kirk McDonald escribió:
> 
> Before I started mucking around with it, CeleriD simply took all of the 
> code files needed, jammed them all on a command line with the 
> appropriate options (-c, mostly), and compiled them. Then it linked them 
> together. Fine.
> 
> When adding gdc support, the first thing I realised was that gdc (and 
> gcc) doesn't support DMD's -od option, and distutils requires that all 
> of the object files be placed in a specific directory. My first stab at 
> getting around this was to compile the sources one at a time, specifying 
> the location of the object files one by one.
> 
> When I tested this with DMD, it compiled but did not link. It complained 
> of undefined symbols. The symbols given were the big long mangles of 
> template instances. I am still not sure why this happened. I fixed it by 
> putting it back the way it was, as a single call to the compiler.
> 
> To get rid of the need for the -od option, I changed the current working 
> directory to where the object files needed to go, and specified the full 
> path to each of the source files. By not specifying a location, the 
> object files just get dumped in the cwd. Problem solved.
> 

Not too helpful, but anyway: if you call gdc as "gdmd", it accepts dmd's 
parameters (instead of gcc's). gdmd is just a wrapper script.

-- 
Carlos Santander Bernal
August 09, 2006
Re: Pyd on Linux building woes
Carlos Santander wrote:
> Kirk McDonald escribió:
> 
>>
>> Before I started mucking around with it, CeleriD simply took all of 
>> the code files needed, jammed them all on a command line with the 
>> appropriate options (-c, mostly), and compiled them. Then it linked 
>> them together. Fine.
>>
>> When adding gdc support, the first thing I realised was that gdc (and 
>> gcc) doesn't support DMD's -od option, and distutils requires that all 
>> of the object files be placed in a specific directory. My first stab 
>> at getting around this was to compile the sources one at a time, 
>> specifying the location of the object files one by one.
>>
>> When I tested this with DMD, it compiled but did not link. It 
>> complained of undefined symbols. The symbols given were the big long 
>> mangles of template instances. I am still not sure why this happened. 
>> I fixed it by putting it back the way it was, as a single call to the 
>> compiler.
>>
>> To get rid of the need for the -od option, I changed the current 
>> working directory to where the object files needed to go, and 
>> specified the full path to each of the source files. By not specifying 
>> a location, the object files just get dumped in the cwd. Problem solved.
>>
> 
> Not too helpful, but anyway: if you call gdc as "gdmd", it accepts dmd's 
> parameters (instead of gcc's). gdmd is just a wrapper script.
> 

I'm well aware of gdmd, but I have chosen to not use it. It is a perl 
script, and will not typically work on Windows. I eventually want to 
support using GDC on Windows.

-- 
Kirk McDonald
Pyd: Wrapping Python with D
http://dsource.org/projects/pyd/wiki
Top | Discussion index | About this forum | D home