Brad and I were talking about some D code that needed openssl support, when we ran into the same old problem:

No D files corresponding to the openssl C .h files.

It's not that these are a big problem to create, it's just that they are not done, and it tends to turn off people from using D. D is binary API compatible with C, but only with a corresponding D import file. This, out of the box, makes D *harder* to use than C.

Lots of people roll their own, but that work is hard to find and haphazard.

This problem keeps coming up again and again.

So I propose creating, on github.com/D-Programming-Language, a new repository called CAPI.

The CAPI Manifesto
------------------

CAPI is a collection of C header files to publicly available C libraries
and their translations to D. The idea is that if, in C, to interface to a library
one would write:

   #include "foo.h"

then the corresponding D code would look like:

   import foo;

Each C .h file would have a corresponding .d file. Each C directory would
have a corresponding D directory, for example:

   #include "bar/foo.h"   // C

   import bar.foo; // D

The top level directory of each library will have two subdirectories:

   C/
   D/

and there will be a one-to-one correspondence of files and directory structure
between them.

The D import files will be a rote translation of the corresponding C .h file.
No attempt will be made to fix, improve, or extend the C api. No attempt will
be made to duplicate the C documentation, or replace it in any way. There
will be no unittests. Every effort will be made to avoid needing any D specific
binary files.

When an updated version of the C header files becomes available, those will
get checked into the C subdirectory tree, and then the corresponding D files
will get updated.

Version tags used must match the version tags used by the C API files.

The license used for the D versions should match the C ones, as they are a
derived work.

On 10/16/11 11:02 PM, Walter Bright wrote:
> Brad and I were talking about some D code that needed openssl support,
> when we ran into the same old problem:
>
> No D files corresponding to the openssl C .h files.
>
> It's not that these are a big problem to create, it's just that they are
> not done, and it tends to turn off people from using D. D is binary API
> compatible with C, but only with a corresponding D import file. This,
> out of the box, makes D *harder* to use than C.
>
> Lots of people roll their own, but that work is hard to find and haphazard.
>
> This problem keeps coming up again and again.
>
> So I propose creating, on github.com/D-Programming-Language, a new
> repository called CAPI.

So you would put every interface to every possible C code there?

In Ruby if you want to have very efficient code you'd implement it as C extensions. For that, you create wrappers in Ruby for C. Now, big part of the standard library has extensions for the most needed things. Everything else, like bindings to an efficient xml parser, are made by different people that public them as gems. Having a public gem repository it's really easy to find bindings for whatever you want. They don't need to be part of the standard library. And it wouldn't make sense, having so much functionality out there available as C code.

So I'd suggest having D headers for the most common things in phobos and focusing on a tool like rubygems. It would give such a big boost to the language.

I also can't imagine how that big repository would work. You'd copy the remote file locally? What if that file gets fixes? You'd copy it again? Or maybe you'd git checkout everything from that repository locally and synchronize it from time to time, with the chance of breaking existing code...

Having "gems" and versioning them should make all these problems disappear. Maybe there is an openssl header in D. The problem is that there might be many, and they don't know each other, and google is a maze to find such things.

Overall, I think that it sounds like a good idea, but I'd suggest that we namespace stuff. So, instead of

import foo;

you'd get something like

import capi.foo;

Otherwise, we unnecessarily increase the odds of module names conflicting with modules in other projects.

Also, if we're putting it up on github, we may want to come up with a cooler name than CAPI (though it _is_ right to the point). There's already a user with the name CAPI ( https://github.com/capi ), so aside from whether CAPI is a good name in its own right or not, that might cause problems. Maybe pick something suitably Greek or Roman, given that we already have Phobos? The name of one of the Titans would be fitting given that it relates to C (on the theory that the Greek gods are the successors of the Titans and D's standard library is named after a Greek god).

It also brings up the question of who is going to manage this project. Someone (or preferrably, a group of someones) is going to have to have commit access in order to merge in pull requests. Do want to just start off with that being the same group of folks with Phobos commit access and grow it from there as appropriate? We certainly wouldn't want to insist that the two groups be the same, but it does seem like a good place to start from.

- Jonathan M Davis

On Mon, 17 Oct 2011 05:02:52 +0300, Walter Bright <newshound2@digitalmars.com> wrote:

> So I propose creating, on github.com/D-Programming-Language, a new repository called CAPI.

Questions:

1) Will these be distributed with DMD? The bar to generate D import modules from C headers isn't much higher than having to find and download headers from the Internet.

2) This isn't too different to already existing projects. The bindings project on dsource ( http://dsource.org/projects/bindings ) already has bindings for various libraries. I can't say much about the project guidelines, though.

3) You suggest to place each library in its own directory, with C and D headers as subdirectories. This means that the user will still need to edit the import search path when using a new library. Is it realistic to put all D files in the same directory? (Perhaps do this only for libraries for which we don't expect name collisions?)

4) "Every effort will be made to avoid needing any D specific binary files." - What about import libraries?

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On Mon, 17 Oct 2011 05:51:57 +0300, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> Also, if we're putting it up on github, we may want to come up with a cooler
> name than CAPI (though it _is_ right to the point). There's already a user
> with the name CAPI ( https://github.com/capi ), so aside from whether CAPI is
> a good name in its own right or not, that might cause problems. Maybe pick
> something suitably Greek or Roman, given that we already have Phobos? The name
> of one of the Titans would be fitting given that it relates to C (on the theory
> that the Greek gods are the successors of the Titans and D's standard library
> is named after a Greek god).

I think the name's fine. I don't see "<4-letter word> is already taken" as a valid argument, considering the huge number of users. There's a GitHub user called "phobos" and one called "tools", etc.

Also, I'm pretty sure the D standard library is named after Mars's moon, considering that D was originally named "Mars Programming Language" (after the company name, "Digital Mars").

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On 10/16/2011 8:17 PM, Vladimir Panteleev wrote:
> I think the name's fine. I don't see "<4-letter word> is already taken" as a
> valid argument, considering the huge number of users. There's a GitHub user
> called "phobos" and one called "tools", etc.
>
> Also, I'm pretty sure the D standard library is named after Mars's moon,
> considering that D was originally named "Mars Programming Language" (after the
> company name, "Digital Mars").

I thought maybe Diemos.

Is this repo intended for use by users or library writers? I don't think encouraging writing C-style code in D is a good idea, that's why I'm asking.

On 10/16/2011 8:10 PM, Vladimir Panteleev wrote:
> 1) Will these be distributed with DMD? The bar to generate D import modules from
> C headers isn't much higher than having to find and download headers from the
> Internet.

I was thinking, no at this time. I suspect it may grow to be quite large, and would become rather onerous. I also don't want to tie it to the DMD release cycle.


> 2) This isn't too different to already existing projects. The bindings project
> on dsource ( http://dsource.org/projects/bindings ) already has bindings for
> various libraries. I can't say much about the project guidelines, though.

The bindings project is a great resource, though it seems a little disorganized. We've had great success with github, meaning it seems to be very good at encouraging community participation.


> 3) You suggest to place each library in its own directory, with C and D headers
> as subdirectories. This means that the user will still need to edit the import
> search path when using a new library.

Yes.

> Is it realistic to put all D files in the
> same directory? (Perhaps do this only for libraries for which we don't expect
> name collisions?)

Hmm, you're right. Perhaps openssl.whatever.


> 4) "Every effort will be made to avoid needing any D specific binary files." -
> What about import libraries?

What do you mean by import libraries? Do you mean Windows DLL import libraries? Those would be supplied by whoever supplied the C library. D can access those directly.

CAPI should be interface source code only library; the D equivalent of #include.

On 10/16/2011 7:18 PM, Ary Manzana wrote:
> So you would put every interface to every possible C code there?

No. It's simply a collection point for interfaces to publicly available C libraries. The place to go to see if someone has already written what you're looking for. Its mission is also narrowly focused on simply being a way to access existing C libraries from D in the most straightforward way possible.


> In Ruby if you want to have very efficient code you'd implement it as C
> extensions. For that, you create wrappers in Ruby for C. Now, big part of the
> standard library has extensions for the most needed things. Everything else,
> like bindings to an efficient xml parser, are made by different people that
> public them as gems. Having a public gem repository it's really easy to find
> bindings for whatever you want. They don't need to be part of the standard
> library. And it wouldn't make sense, having so much functionality out there
> available as C code.

Right.

>
> So I'd suggest having D headers for the most common things in phobos

Yes.

> and focusing on a tool like rubygems. It would give such a big boost to the language.

I think github is adequate for now to be the tool.


> I also can't imagine how that big repository would work. You'd copy the remote
> file locally?

That'll work, or you can use the github 'clone' feature.

> What if that file gets fixes? You'd copy it again? Or maybe you'd
> git checkout everything from that repository locally and synchronize it from
> time to time, with the chance of breaking existing code...
> Having "gems" and versioning them should make all these problems disappear.

Github has extensive versioning and branching support. This shouldn't be an issue.

> Maybe there is an openssl header in D. The problem is that there might be many,
> and they don't know each other, and google is a maze to find such things.

Exactly. And maybe there are several openssl D headers, each of which is half-assed in a different way. With a central library for them, we can improve them globally rather than piecemeal and randomly.

On 10/16/2011 7:51 PM, Jonathan M Davis wrote:
> Overall, I think that it sounds like a good idea, but I'd suggest that we
> namespace stuff. So, instead of
>
> import foo;
>
> you'd get something like
>
> import capi.foo;
>
> Otherwise, we unnecessarily increase the odds of module names conflicting with
> modules in other projects.

Perhaps you're right.


> Also, if we're putting it up on github, we may want to come up with a cooler
> name than CAPI (though it _is_ right to the point). There's already a user
> with the name CAPI ( https://github.com/capi ), so aside from whether CAPI is
> a good name in its own right or not, that might cause problems. Maybe pick
> something suitably Greek or Roman, given that we already have Phobos? The name
> of one of the Titans would be fitting given that it relates to C (on the theory
> that the Greek gods are the successors of the Titans and D's standard library
> is named after a Greek god).

Unless CAPI is trademarked, I think we're in good shape.


> It also brings up the question of who is going to manage this project. Someone
> (or preferrably, a group of someones) is going to have to have commit access
> in order to merge in pull requests. Do want to just start off with that being
> the same group of folks with Phobos commit access and grow it from there as
> appropriate? We certainly wouldn't want to insist that the two groups be the
> same, but it does seem like a good place to start from.

I figure initially the same team members as are on the phobos team, but it would be a separate team with its own member list.