On 10/16/2011 9:02 PM, Andrej Mitrovic wrote:
> Is this repo intended for use by users or library writers?

Both.

> I don't
> think encouraging writing C-style code in D is a good idea, that's why
> I'm asking.

D's ability to directly call any C code is a huge advantage, but one we've underutilized because of a lack of interface imports to the great mass of popular C libraries.

For example, are we going to reinvent openssl? libcurl? imagemajick? No way.

Please excuse my ignorance,
but several types in D do not currently translate well into C.

For example, strings in D are not '\0' terminated, which breaks with C. This is not usually a problem, and it's easy to wrap the function in the translated header file to automate that particular process.

It seems that your proposal would disallow this particular example. The translation code would have to be somewhere, what is the argument against allowing it?

In my (admittedly ignorant) opinion, it seems that allowing the automatic
translation of D types to C types
would fit very well, otherwise you'll have to add them yourself every time
you wish to call C.

Forgive me If I have misunderstood something.

I think there might be a few tricks to improve the C API without
adding any new code.
For example, replace by-pointer parameter declarations with _out_
parameters when applicable (the underlying function signature is the
same), replace const parameters with in parameters, etc.
This won't change the C API a single bit (won't even add new code),
but will vastly improve readability and sometimes safety of the API.
In other cases, some minor additions could be made, for example:
libjpeg provides API to register error handlers, instead of setting
errno and such. Those kind of situations could be used to throw
exceptions. It only takes a static this() and a few lines of code.
In case those kind of modifications/additions are made, there could be
a standard way to disable them and use the original version.

On Mon, Oct 17, 2011 at 6:02 AM, Walter Bright <newshound2@digitalmars.com> wrote:
> Brad and I were talking about some D code that needed openssl support, when we ran into the same old problem:
>
> No D files corresponding to the openssl C .h files.
>
> It's not that these are a big problem to create, it's just that they are not done, and it tends to turn off people from using D. D is binary API compatible with C, but only with a corresponding D import file. This, out of the box, makes D *harder* to use than C.
>
> Lots of people roll their own, but that work is hard to find and haphazard.
>
> This problem keeps coming up again and again.
>
> So I propose creating, on github.com/D-Programming-Language, a new repository called CAPI.
>
> The CAPI Manifesto
> ------------------
>
> CAPI is a collection of C header files to publicly available C libraries
> and their translations to D. The idea is that if, in C, to interface to a
> library
> one would write:
>
>   #include "foo.h"
>
> then the corresponding D code would look like:
>
>   import foo;
>
> Each C .h file would have a corresponding .d file. Each C directory would have a corresponding D directory, for example:
>
>   #include "bar/foo.h"   // C
>
>   import bar.foo; // D
>
> The top level directory of each library will have two subdirectories:
>
>   C/
>   D/
>
> and there will be a one-to-one correspondence of files and directory
> structure
> between them.
>
> The D import files will be a rote translation of the corresponding C .h
> file.
> No attempt will be made to fix, improve, or extend the C api. No attempt
> will
> be made to duplicate the C documentation, or replace it in any way. There
> will be no unittests. Every effort will be made to avoid needing any D
> specific
> binary files.
>
> When an updated version of the C header files becomes available, those will get checked into the C subdirectory tree, and then the corresponding D files will get updated.
>
> Version tags used must match the version tags used by the C API files.
>
> The license used for the D versions should match the C ones, as they are a derived work.
>

OR

Maybe parallel to the CAPI, there could be the D-ified version of it,
that will be developed after the original CAPI.
This is, of course, not as urgent a the CAPI itself, but it would be
very useful to gradually help users get rid of unnecessarily dangerous
code.

On Mon, Oct 17, 2011 at 9:24 AM, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:
> I think there might be a few tricks to improve the C API without
> adding any new code.
> For example, replace by-pointer parameter declarations with _out_
> parameters when applicable (the underlying function signature is the
> same), replace const parameters with in parameters, etc.
> This won't change the C API a single bit (won't even add new code),
> but will vastly improve readability and sometimes safety of the API.
> In other cases, some minor additions could be made, for example:
> libjpeg provides API to register error handlers, instead of setting
> errno and such. Those kind of situations could be used to throw
> exceptions. It only takes a static this() and a few lines of code.
> In case those kind of modifications/additions are made, there could be
> a standard way to disable them and use the original version.
>
> On Mon, Oct 17, 2011 at 6:02 AM, Walter Bright <newshound2@digitalmars.com> wrote:
>> Brad and I were talking about some D code that needed openssl support, when we ran into the same old problem:
>>
>> No D files corresponding to the openssl C .h files.
>>
>> It's not that these are a big problem to create, it's just that they are not done, and it tends to turn off people from using D. D is binary API compatible with C, but only with a corresponding D import file. This, out of the box, makes D *harder* to use than C.
>>
>> Lots of people roll their own, but that work is hard to find and haphazard.
>>
>> This problem keeps coming up again and again.
>>
>> So I propose creating, on github.com/D-Programming-Language, a new repository called CAPI.
>>
>> The CAPI Manifesto
>> ------------------
>>
>> CAPI is a collection of C header files to publicly available C libraries
>> and their translations to D. The idea is that if, in C, to interface to a
>> library
>> one would write:
>>
>>   #include "foo.h"
>>
>> then the corresponding D code would look like:
>>
>>   import foo;
>>
>> Each C .h file would have a corresponding .d file. Each C directory would have a corresponding D directory, for example:
>>
>>   #include "bar/foo.h"   // C
>>
>>   import bar.foo; // D
>>
>> The top level directory of each library will have two subdirectories:
>>
>>   C/
>>   D/
>>
>> and there will be a one-to-one correspondence of files and directory
>> structure
>> between them.
>>
>> The D import files will be a rote translation of the corresponding C .h
>> file.
>> No attempt will be made to fix, improve, or extend the C api. No attempt
>> will
>> be made to duplicate the C documentation, or replace it in any way. There
>> will be no unittests. Every effort will be made to avoid needing any D
>> specific
>> binary files.
>>
>> When an updated version of the C header files becomes available, those will get checked into the C subdirectory tree, and then the corresponding D files will get updated.
>>
>> Version tags used must match the version tags used by the C API files.
>>
>> The license used for the D versions should match the C ones, as they are a derived work.
>>
>

On 10/16/2011 9:28 PM, Jude Young wrote:
> Please excuse my ignorance,
> but several types in D do not currently translate well into C.
>
> For example, strings in D are not '\0' terminated, which breaks with C.

This is incorrect. String literals in D are 0 terminated, as in C. C has a convention that strings are 0 terminated, but it isn't part of the core language. In D, you can terminate a string with 0 if you want to, or not.

> This is not usually a problem, and it's easy to wrap the function in the
> translated header file to automate that particular process.

This would fall under attempting to fix the api, which would be beyond the scope of the library.


> It seems that your proposal would disallow this particular example.
> The translation code would have to be somewhere, what is the argument against
> allowing it?

D access to the C API should be direct, and not include hidden costs like translation layers.


> In my (admittedly ignorant) opinion, it seems that allowing the automatic
> translation of D types to C types
> would fit very well, otherwise you'll have to add them yourself every time you
> wish to call C.

C doesn't actually even have a 'string' type.

On 10/16/2011 10:24 PM, Gor Gyolchanyan wrote:
> I think there might be a few tricks to improve the C API without
> adding any new code.
> For example, replace by-pointer parameter declarations with _out_
> parameters when applicable (the underlying function signature is the
> same), replace const parameters with in parameters, etc.
> This won't change the C API a single bit (won't even add new code),
> but will vastly improve readability and sometimes safety of the API.
> In other cases, some minor additions could be made, for example:
> libjpeg provides API to register error handlers, instead of setting
> errno and such. Those kind of situations could be used to throw
> exceptions. It only takes a static this() and a few lines of code.
> In case those kind of modifications/additions are made, there could be
> a standard way to disable them and use the original version.

If you want to add a layer on top of the C API, that would be fine. std.zlib is an example of that.

But the idea of CAPI is NOT to add a layer. Not fix, extend, refactor, improve, etc.

Just the thinnest possible direct calls to the C API. Any improvements, fixes, whatever, should be a separate project.

I know the urge to do these fixes can be overpowering, but they end badly every time. It's like trying to mix the language's lexer up with the semantic analysis :-)

They really are better off being separate and distinct.

For one thing, it makes the inevitable maintenance *FAR* easier, as those C APIs will change. Updating the corresponding D module becomes simple then - just a line by line comparison and tweaking. There would be no tearing of hair and rending of garments.

For another it means you'll have to regenerate the C documentation, but with all the changes you made. Then, as the C guys improve their documentation, your layer falls behind, gets neglected, and finally sucks in comparison.

On Mon, 17 Oct 2011 08:24:26 +0300, Gor Gyolchanyan <gor.f.gyolchanyan@gmail.com> wrote:

> For example, replace by-pointer parameter declarations with _out_
> parameters when applicable (the underlying function signature is the
> same)

Note that this would make it nearly impossible to pass null pointers. A human would need to decide if a null pointer can be specified, or the converter would need to be aware of proprietary extensions which specify such things (e.g. Microsoft's __in).

-- 
Best regards,
 Vladimir                            mailto:vladimir@thecybershadow.net

On 10/16/2011 11:21 PM, Vladimir Panteleev wrote:
> On Mon, 17 Oct 2011 08:24:26 +0300, Gor Gyolchanyan
> <gor.f.gyolchanyan@gmail.com> wrote:
>
>> For example, replace by-pointer parameter declarations with _out_
>> parameters when applicable (the underlying function signature is the
>> same)
>
> Note that this would make it nearly impossible to pass null pointers. A human
> would need to decide if a null pointer can be specified, or the converter would
> need to be aware of proprietary extensions which specify such things (e.g.
> Microsoft's __in).


Right. You could also annotate C API functions with pure, @safe, etc., but you'd have to be very careful that those functions actually were that way, and would not violate those attributes in the future.

Fair enough, thanks for the time.
I found a thin D binding to ncurses, and 'fixed' it to work with D2.
If y'all get this up and running, this code will probably be better than
starting from scratch.

In any case, I think that CAPI is exactly the type of thing that D needs
going forward.
I really hope that CAPI gets good support.

On Mon, Oct 17, 2011 at 12:55 AM, Walter Bright <newshound2@digitalmars.com>wrote:

> On 10/16/2011 9:28 PM, Jude Young wrote:
>
>> Please excuse my ignorance,
>> but several types in D do not currently translate well into C.
>>
>> For example, strings in D are not '\0' terminated, which breaks with C.
>>
>
> This is incorrect. String literals in D are 0 terminated, as in C. C has a convention that strings are 0 terminated, but it isn't part of the core language. In D, you can terminate a string with 0 if you want to, or not.
>
>
>  This is not usually a problem, and it's easy to wrap the function in the
>> translated header file to automate that particular process.
>>
>
> This would fall under attempting to fix the api, which would be beyond the scope of the library.
>
>
>
>  It seems that your proposal would disallow this particular example.
>> The translation code would have to be somewhere, what is the argument
>> against
>> allowing it?
>>
>
> D access to the C API should be direct, and not include hidden costs like translation layers.
>
>
>
>  In my (admittedly ignorant) opinion, it seems that allowing the automatic
>> translation of D types to C types
>> would fit very well, otherwise you'll have to add them yourself every time
>> you
>> wish to call C.
>>
>
> C doesn't actually even have a 'string' type.
>

On 2011-10-17 04:18, Ary Manzana wrote:
> On 10/16/11 11:02 PM, Walter Bright wrote:
>> Brad and I were talking about some D code that needed openssl support,
>> when we ran into the same old problem:
>>
>> No D files corresponding to the openssl C .h files.
>>
>> It's not that these are a big problem to create, it's just that they are
>> not done, and it tends to turn off people from using D. D is binary API
>> compatible with C, but only with a corresponding D import file. This,
>> out of the box, makes D *harder* to use than C.
>>
>> Lots of people roll their own, but that work is hard to find and
>> haphazard.
>>
>> This problem keeps coming up again and again.
>>
>> So I propose creating, on github.com/D-Programming-Language, a new
>> repository called CAPI.
>
> So you would put every interface to every possible C code there?
>
> In Ruby if you want to have very efficient code you'd implement it as C
> extensions. For that, you create wrappers in Ruby for C. Now, big part
> of the standard library has extensions for the most needed things.
> Everything else, like bindings to an efficient xml parser, are made by
> different people that public them as gems. Having a public gem
> repository it's really easy to find bindings for whatever you want. They
> don't need to be part of the standard library. And it wouldn't make
> sense, having so much functionality out there available as C code.
>
> So I'd suggest having D headers for the most common things in phobos and
> focusing on a tool like rubygems. It would give such a big boost to the
> language.
>
> I also can't imagine how that big repository would work. You'd copy the
> remote file locally? What if that file gets fixes? You'd copy it again?
> Or maybe you'd git checkout everything from that repository locally and
> synchronize it from time to time, with the chance of breaking existing
> code...
>
> Having "gems" and versioning them should make all these problems
> disappear. Maybe there is an openssl header in D. The problem is that
> there might be many, and they don't know each other, and google is a
> maze to find such things.

Already working on a package manager for D:

https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D
https://github.com/jacob-carlborg/orbit/

-- 
/Jacob Carlborg