View mode: basic / threaded / horizontal-split · Log in · Help
October 15, 2012
More automated interfacing of D with C codebases
This will be my first posting here, but I've been obsessed with D 
since before there was a D2 standard. Despite that rather long 
period of obsession, it's only just now that I've started 
seriously using D as I attempt to code-up some POC projects to 
demonstrate that the language is mature enough for serious 
development, or at least the kind of development we're doing, 
anyway.

So, that out of the way, I suppose I should apologize in advance 
for what must amount to a pretty dumb question, since I truly 
couldn't have been the first to think of this: with regard to 
interfacing D with C libraries, why would we require a project 
like Deimos which ensures that each codebase is ported, almost 
entirely, by hand? Theoretically, shouldn't there be some kind of 
autotools-like method to establish something like, config.d.in 
which is then used via abstractions to properly configure the 
rest of the D wrapper? Obviously, D doesn't have a preprocessor, 
but couldn't one synthesize that with a series of judicious 
version() blocks? Then, whenever someone goes to build the 
application on their system, you'd have something to a series of 
autoconf-like tests to fill in the proper sections of of 
config.d.in and get out a config.d suitable for their system.

I'm really sorry if there's some stupidly obvious reason this 
couldn't work and I'm just not seeing it, or if this is some n00b 
question you get all the time.
October 15, 2012
Re: More automated interfacing of D with C codebases
On 2012-10-15 13:51, CapitalLetterC wrote:
> This will be my first posting here, but I've been obsessed with D since
> before there was a D2 standard. Despite that rather long period of
> obsession, it's only just now that I've started seriously using D as I
> attempt to code-up some POC projects to demonstrate that the language is
> mature enough for serious development, or at least the kind of
> development we're doing, anyway.
>
> So, that out of the way, I suppose I should apologize in advance for
> what must amount to a pretty dumb question, since I truly couldn't have
> been the first to think of this: with regard to interfacing D with C
> libraries, why would we require a project like Deimos which ensures that
> each codebase is ported, almost entirely, by hand? Theoretically,
> shouldn't there be some kind of autotools-like method to establish
> something like, config.d.in which is then used via abstractions to
> properly configure the rest of the D wrapper? Obviously, D doesn't have
> a preprocessor, but couldn't one synthesize that with a series of
> judicious version() blocks? Then, whenever someone goes to build the
> application on their system, you'd have something to a series of
> autoconf-like tests to fill in the proper sections of of config.d.in and
> get out a config.d suitable for their system.
>
> I'm really sorry if there's some stupidly obvious reason this couldn't
> work and I'm just not seeing it, or if this is some n00b question you
> get all the time.

The header files need to be ported so D can find the declarations of the 
functions and types. There are a couple of tools that can do most of the 
work, but not completely. Macros are specially hard to port automatically.

https://github.com/jacob-carlborg/dstep

-- 
/Jacob Carlborg
October 15, 2012
Re: More automated interfacing of D with C codebases
On Monday, 15 October 2012 at 12:54:51 UTC, Jacob Carlborg wrote:
> On 2012-10-15 13:51, CapitalLetterC wrote:
>> This will be my first posting here, but I've been obsessed 
>> with D since
>> before there was a D2 standard. Despite that rather long 
>> period of
>> obsession, it's only just now that I've started seriously 
>> using D as I
>> attempt to code-up some POC projects to demonstrate that the 
>> language is
>> mature enough for serious development, or at least the kind of
>> development we're doing, anyway.
>>
>> So, that out of the way, I suppose I should apologize in 
>> advance for
>> what must amount to a pretty dumb question, since I truly 
>> couldn't have
>> been the first to think of this: with regard to interfacing D 
>> with C
>> libraries, why would we require a project like Deimos which 
>> ensures that
>> each codebase is ported, almost entirely, by hand? 
>> Theoretically,
>> shouldn't there be some kind of autotools-like method to 
>> establish
>> something like, config.d.in which is then used via 
>> abstractions to
>> properly configure the rest of the D wrapper? Obviously, D 
>> doesn't have
>> a preprocessor, but couldn't one synthesize that with a series 
>> of
>> judicious version() blocks? Then, whenever someone goes to 
>> build the
>> application on their system, you'd have something to a series 
>> of
>> autoconf-like tests to fill in the proper sections of of 
>> config.d.in and
>> get out a config.d suitable for their system.
>>
>> I'm really sorry if there's some stupidly obvious reason this 
>> couldn't
>> work and I'm just not seeing it, or if this is some n00b 
>> question you
>> get all the time.
>
> The header files need to be ported so D can find the 
> declarations of the functions and types. There are a couple of 
> tools that can do most of the work, but not completely. Macros 
> are specially hard to port automatically.
>
> https://github.com/jacob-carlborg/dstep

Actually, I understand perfectly well why the headers need to be 
ported to D code, what I was more wondering is if anybody knew of 
or had ever tried to use some kind of build system to automate 
the process to the greatest extent possible. So, for example, as 
a result of the configuration process certain features may be 
implicitly or explicitly enabled/disabled, without probing the 
individual user's configuration, you wouldn't know that, so how 
does one maximize portability, in that case? If you simply go to 
support the maximal set of features, you may wind up with a bunch 
of codepaths that go nowhere, likewise, if you choose only to 
support the minimal set of functionality, you may not wind up 
with code that's not useful to anyone, yourself included. 
Granted, I hadn't thought of the difficulty in porting things 
like macros, but I'd imagine that, in the absolute worst case, 
you could synthesize their behavior with templates.

But there are all sorts of issues in that same vein, like how 
certain structs get put together, what types are aliased with 
what, etc. That's why I was wondering if you could handle this 
analogously to how C does, with a config file that gets setup by 
a kind of configuration system.
October 15, 2012
Re: More automated interfacing of D with C codebases
On 2012-10-15 19:20, CapitalLetterC wrote:

> But there are all sorts of issues in that same vein, like how certain
> structs get put together, what types are aliased with what, etc. That's
> why I was wondering if you could handle this analogously to how C does,
> with a config file that gets setup by a kind of configuration system.

I think that would be possible but I haven't seen anyone program in D 
like that.

-- 
/Jacob Carlborg
October 15, 2012
Re: More automated interfacing of D with C codebases
On Monday, 15 October 2012 at 20:09:03 UTC, Jacob Carlborg wrote:
> I think that would be possible but I haven't seen anyone 
> program in D like that.

One of the primary reasons I was concerned that there might be 
some serious flaw I'm overlooking. Oh well, that's a good enough 
excuse for me to begin experimentation. Thanks! :D
October 21, 2012
Re: More automated interfacing of D with C codebases
I don't think this is exactly what you want either, but it's 
certainly a lot easier than a manual process:

http://gentoo.com/di/mkconfig.html
(On Freecode) https://freecode.com/projects/mkconfig

I just now released a new version.  Freecode will be updated 
within 24 hours.
Check the examples/dtcltk/ directory.

There's also a simpler example within the D/ subdirectory in:
  http://gentoo.com/di/

Macros have to have their signature specified. e.g.:

    cmacro Tcl_InitMemory tcl.h void 'Tcl_Interp *'
    cmacro ScreenOfDisplay X11/Xlib.h 'Screen *' 'Display' int

 -- Brad
October 21, 2012
Re: More automated interfacing of D with C codebases
Manually porting of C/C++ libraries shouldn't be the way to go (a
major problem being when said library gets updated, human errors,
porting to different architectures etc).

I've had good experience using SWIG to automate the job for me,
and used it successfully to port the following libraries to D:

* ncurses
* opencv
* sfml
to name a few.

I believe this is the way forward, as opposed to maintaining
(necessary stale) ports such as Deimos.

Swig isn't perfect however, and sometimes will bail out on
constructs such as:
C++ classes with multiple inheritance, or a few obscure C or C++
constructs. Even in that case, I find it much easier to tweak the
swig interface file to get desired output rather than undertake
the huge manual conversion task. For sfml, the interface file is
just about 40 lines for example.

Improving such kind of automated tools is where we should be
concentrating efforts, not on maintaining deimos. One such
improvement I made was to allow conversion of C++ templates to D
templates (works in large majority of cases), which makes the
library much easier to use than using name mangling schemes. More
improvements would be welcome (eg support for static library
generation as opposed to dynamic load libraries), reducing the
need for tweaking swig interface files etc.
October 21, 2012
Re: More automated interfacing of D with C codebases
On Sunday, 21 October 2012 at 19:45:53 UTC, timotheecour wrote:
> I've had good experience using SWIG to automate the job for me,
> and used it successfully to port the following libraries to D:

Yes.  SWIG should be a lot faster than my tool.
Also I don't have any units to handle C++ code.

Completely agree. It has to be automated.

> Swig isn't perfect however, and sometimes will bail out on
> constructs such as:
> C++ classes with multiple inheritance, or a few obscure C or C++
> constructs. Even in that case, I find it much easier to tweak

If I recall, I ran into some obscure C constructs in the Tcl/Tk/X 
headers that I have never seen before.  I believe I had to 
various aliases to my "interface file" to handle those.  Also use 
of D reserved words had to be handled:

  substitute new new_
  substitute class class_
  substitute 'function;' 'function_;'

And some things just plain won't work in D:

  # not valid D code.
  substitute '_XPrivate .private9, .private10;' \
            '_XPrivate * private9, private10;'
October 22, 2012
Re: More automated interfacing of D with C codebases
On 2012-10-21 21:45, timotheecour wrote:
> Manually porting of C/C++ libraries shouldn't be the way to go (a
> major problem being when said library gets updated, human errors,
> porting to different architectures etc).

SWIG requires some kind of interface files, which I assume one must have 
to manually write. Then what's the difference?

> I've had good experience using SWIG to automate the job for me,
> and used it successfully to port the following libraries to D:
>
> * ncurses
> * opencv
> * sfml
> to name a few.
>
> I believe this is the way forward, as opposed to maintaining
> (necessary stale) ports such as Deimos.
>
> Swig isn't perfect however, and sometimes will bail out on
> constructs such as:
> C++ classes with multiple inheritance, or a few obscure C or C++
> constructs. Even in that case, I find it much easier to tweak the
> swig interface file to get desired output rather than undertake
> the huge manual conversion task. For sfml, the interface file is
> just about 40 lines for example.
>
> Improving such kind of automated tools is where we should be
> concentrating efforts, not on maintaining deimos. One such
> improvement I made was to allow conversion of C++ templates to D
> templates (works in large majority of cases), which makes the
> library much easier to use than using name mangling schemes. More
> improvements would be welcome (eg support for static library
> generation as opposed to dynamic load libraries), reducing the
> need for tweaking swig interface files etc.

I don't think SWIG is the right approach because:

* You have to create these interface files
* It seem to not handle all code
* No support for Objective-C

BTW, how does SWIG handle macros and documentation?

-- 
/Jacob Carlborg
October 22, 2012
Re: More automated interfacing of D with C codebases
> SWIG requires some kind of interface files, which I assume one 
> must have to manually write. Then what's the difference?

These can be as small as:


sfml.i:
%{
#include <SFML/Audio.hpp>
%}
%include SFML/Config.hpp
%include SFML/Audio/Export.hpp

In practice, some additional tweaks may be needed to support 
certain constructs like multiple inheritance, etc, but nothing 
daunting (~ 50 lines of code for my sfml interface file)

> I don't think SWIG is the right approach because:
> * You have to create these interface files

=>
see above. Interface files can be very small.

> * It seem to not handle all code
I was still able to port large libraries like sfml and opencv, 
using tweaks to the interface files and enhancing swig to support 
C++template to D template conversion. There's a small number of 
cases where swig currently chokes, some of which could be easily 
fixed if there's enough demand.

> * No support for Objective-C

=>
http://www.swig.org/Doc1.1/HTML/SWIG.html#n7
One of SWIG's most recent additions is support for Objective-C 
parsing. This is currently an experimental feature that may be 
improved in future releases.

> BTW, how does SWIG handle macros and documentation?
You'd have to write a C/C++ function that does what the macro 
does in the interface file, which would in turn be converted to a 
D function.

For exporting C documentation to D, I've asked on stackoverflow: 
http://stackoverflow.com/questions/12413957/export-comments-in-swig-wrapper
I'm not sure it's supported ATM, but that's definitely something 
possible since swig has a full blown C++ parser.
« First   ‹ Prev
1 2 3
Top | Discussion index | About this forum | D home