Hi,

I try to get wrap the "Azure SDK for C" using DPP and have following issue.
Functions, which are actually implemented in C header files will cause
linker errors:

https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91

Example:
AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr, int32_t length, int32_t capacity) {
  return (az_span){ ._internal = { .ptr = ptr, .length = length, .capacity = capacity, }, };
}

Error message:
/tmp/app.o:az_storage_blobs.d:function _D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_storage_blobs7az_span: error: undefined reference to 'az_span_init'

I do not know C well, is this the expected behavior and should I ask the Azure SDK developers to not implement functions within C header files?

Kind regards
André

On Tuesday, 18 February 2020 at 05:41:38 UTC, Andre Pany wrote:
> Hi,
>
> I try to get wrap the "Azure SDK for C" using DPP and have following issue.
> Functions, which are actually implemented in C header files will cause
> linker errors:
>
> https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91
>
> Example:
> AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr, int32_t length, int32_t capacity) {
>   return (az_span){ ._internal = { .ptr = ptr, .length = length, .capacity = capacity, }, };
> }
>
> Error message:
> /tmp/app.o:az_storage_blobs.d:function _D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_storage_blobs7az_span: error: undefined reference to 'az_span_init'
>
> I do not know C well, is this the expected behavior and should I ask the Azure SDK developers to not implement functions within C header files?
>
> Kind regards
> André

I think the problem is that you haven't actually linked in the Azure SDK C library.

Dpp translates the header declarations from C to D, but the actual definitions (function bodies) are not part of the process. The executable code for the function definitions should be inside either a static or dynamic library provided by the SDK.


From the the project's readme file, it looks like they're using CMake as the build system generator (afterwards both make and ninja should be valid choices for building):

mkdir build
cd build
cmake ../
make

In cases like this, it's best to checkout the CMakeLists.txt files of the individual sub project, like this one:

https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/CMakeLists.txt

As you can see, there are several outputs of the build process, among which:

- add_library(az_core ...)
This defines a library named az_core which can produce either a static (.a on Linux, .lib on Windows) or dynamic library file (.so on Linux, .dll on Windows). (If no configuration is specified, I think it's static by default).
So the final file name would be libaz_core.{a,so} on Linux.
For the .c files to be built, a list of include directories must be specified, where the various .h would located (containing function and type declarations). This done like so:
target_include_directories (az_core PUBLIC ...)
The 'PUBLIC' argument to the target_include_directories specifies that if you want to use the library, you need to use the same include directories, as those needed for building it.

- add_executable(az_core_test ..)
This defines an executable build output, which looks is only used for testing, so it's not interesting to us, except that it can serve as an example app using the az_core library.

---

So in summary, if you want to use the az_core library, you need to:
1. Build it
2. Run Dpp like so:

d++ \
  --include-path <specify the same as what was given to target_include_directories>
  <your D file(s) here)
  <path to libaz_core.a>


You will need to repeat the same steps for any other part of the Azure C SDK.


--------

TL;DR
After I went through all those steps I got a similar linker error for az_http_response_init. After looking where is the actual function definition, it turned out that it's not defined in a .c file, but it is an inline function part of a header file.
Searching for az_span_init revealed the same (I could have saved myself some time by reading your message more carefully :D).

So, to answer your original question, the problem is that dpp translates only declarations, not function definitions (such as inline functions like that).

For now, your best course of action is to translate all inline function definition by hand. Since in C inline functions are mostly short and simple functions (a better alternative to macros), hopefully that won't be too much work.

Also, looking at macros like AZ_SPAN_FROM_STR, there's really very little chance that they could be correctly translated automatically. As the things they do are likely not valid even in @system D code (without additional casts), so it's better to write your own D functions by hand anyway.


Here's what I tried:

test.dpp:

#include <az_http.h>
#include <az_span.h>

import std.stdio;

void main()
{
    char[] resp =
        "HTTP/1.2 404 We removed the\tpage!\r\n" ~
        "\r\n" ~
        "But there is somebody. :-)".dup;
    az_span response_span =
    {{
        ptr: cast(ubyte*)resp.ptr,
        length: cast(int)resp.length,
        capacity: cast(int)resp.length
    }};
    az_http_response response;
    az_result result = az_http_response_init(
        &response, response_span);

    writeln(result);
}

d++ --compiler ldmd2 --include-path ./inc test.dpp ./build/libaz_core.a

On Tuesday, 18 February 2020 at 08:32:47 UTC, Petar Kirov [ZombineDev] wrote:
> On Tuesday, 18 February 2020 at 05:41:38 UTC, Andre Pany wrote:
>> Hi,
>>
>> I try to get wrap the "Azure SDK for C" using DPP and have following issue.
>> Functions, which are actually implemented in C header files will cause
>> linker errors:
>>
>> https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91
>>
>> Example:
>> AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr, int32_t length, int32_t capacity) {
>>   return (az_span){ ._internal = { .ptr = ptr, .length = length, .capacity = capacity, }, };
>> }
>>
>> Error message:
>> /tmp/app.o:az_storage_blobs.d:function _D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_storage_blobs7az_span: error: undefined reference to 'az_span_init'
>>
>> I do not know C well, is this the expected behavior and should I ask the Azure SDK developers to not implement functions within C header files?
>>
>> Kind regards
>> André
>
> I think the problem is that you haven't actually linked in the Azure SDK C library.
>
> Dpp translates the header declarations from C to D, but the actual definitions (function bodies) are not part of the process. The executable code for the function definitions should be inside either a static or dynamic library provided by the SDK.
>
>
> From the the project's readme file, it looks like they're using CMake as the build system generator (afterwards both make and ninja should be valid choices for building):
>
> mkdir build
> cd build
> cmake ../
> make
>
> In cases like this, it's best to checkout the CMakeLists.txt files of the individual sub project, like this one:
>
> https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/CMakeLists.txt
>
> As you can see, there are several outputs of the build process, among which:
>
> - add_library(az_core ...)
> This defines a library named az_core which can produce either a static (.a on Linux, .lib on Windows) or dynamic library file (.so on Linux, .dll on Windows). (If no configuration is specified, I think it's static by default).
> So the final file name would be libaz_core.{a,so} on Linux.
> For the .c files to be built, a list of include directories must be specified, where the various .h would located (containing function and type declarations). This done like so:
> target_include_directories (az_core PUBLIC ...)
> The 'PUBLIC' argument to the target_include_directories specifies that if you want to use the library, you need to use the same include directories, as those needed for building it.
>
> - add_executable(az_core_test ..)
> This defines an executable build output, which looks is only used for testing, so it's not interesting to us, except that it can serve as an example app using the az_core library.
>
> ---
>
> So in summary, if you want to use the az_core library, you need to:
> 1. Build it
> 2. Run Dpp like so:
>
> d++ \
>   --include-path <specify the same as what was given to target_include_directories>
>   <your D file(s) here)
>   <path to libaz_core.a>
>
>
> You will need to repeat the same steps for any other part of the Azure C SDK.
>
>
> --------
>
> TL;DR
> After I went through all those steps I got a similar linker error for az_http_response_init. After looking where is the actual function definition, it turned out that it's not defined in a .c file, but it is an inline function part of a header file.
> Searching for az_span_init revealed the same (I could have saved myself some time by reading your message more carefully :D).
>
> So, to answer your original question, the problem is that dpp translates only declarations, not function definitions (such as inline functions like that).
>
> For now, your best course of action is to translate all inline function definition by hand. Since in C inline functions are mostly short and simple functions (a better alternative to macros), hopefully that won't be too much work.
>
> Also, looking at macros like AZ_SPAN_FROM_STR, there's really very little chance that they could be correctly translated automatically. As the things they do are likely not valid even in @system D code (without additional casts), so it's better to write your own D functions by hand anyway.
>
>
> Here's what I tried:
>
> test.dpp:
>
> #include <az_http.h>
> #include <az_span.h>
>
> import std.stdio;
>
> void main()
> {
>     char[] resp =
>         "HTTP/1.2 404 We removed the\tpage!\r\n" ~
>         "\r\n" ~
>         "But there is somebody. :-)".dup;
>     az_span response_span =
>     {{
>         ptr: cast(ubyte*)resp.ptr,
>         length: cast(int)resp.length,
>         capacity: cast(int)resp.length
>     }};
>     az_http_response response;
>     az_result result = az_http_response_init(
>         &response, response_span);
>
>     writeln(result);
> }
>
> d++ --compiler ldmd2 --include-path ./inc test.dpp ./build/libaz_core.a

Hi Petar,

thank you very much for the explanation and the code sample.
Filling the az_span anonymous member was the tricky part,
I thought it would be not possible to do so, but you showed me
the trick.

I will do it like you have proposed but had also already created
a ticket for the Azure SDK developer:
https://github.com/Azure/azure-sdk-for-c/issues/359
There should be a more convenient way to fill a az_span structure.

For reference, here is my dockerfile which does the DPP call and linking:

``` dockerfile
FROM dlang2/ldc-ubuntu:1.20.0 as ldc

RUN apt-get install -y git libssl-dev uuid-dev libcurl4-openssl-dev curl

RUN curl -OL https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh \
    && mkdir /opt/cmake \
    && sh /cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake --skip-license \
    && ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake

RUN git clone https://github.com/Azure/azure-sdk-for-c.git \
    && cd azure-sdk-for-c \
    && git submodule update --init --recursive

RUN cd azure-sdk-for-c \
    && mkdir build \
    && cd build \
    && cmake ../ \
    && make

RUN apt-get install -y clang-9 libclang-9-dev
RUN ln -s /usr/bin/clang-9 /usr/bin/clang
COPY az_storage_blobs.dpp /tmp/

RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- --help

RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- /tmp/az_storage_blobs.dpp \
    --include-path /azure-sdk-for-c/sdk/core/core/inc \
    --include-path /azure-sdk-for-c/sdk/core/core/internal \
    --include-path /azure-sdk-for-c/sdk/storage/blobs/inc \
    --include-path /azure-sdk-for-c/sdk/transport_policies/curl/inc \
    --preprocess-only

ADD blobs_client_example.d /tmp/blobs_client_example.d
RUN  ldc2 /tmp/blobs_client_example.d /tmp/az_storage_blobs.d \
    /azure-sdk-for-c/build/sdk/core/core/libaz_core.a \
    /azure-sdk-for-c/build/sdk/storage/blobs/libaz_storage_blobs.a \
    /azure-sdk-for-c/build/sdk/transport_policies/curl/libaz_curl.a \
    -of=/tmp/app
```

Kind regards
André

On Tuesday, 18 February 2020 at 09:20:08 UTC, Andre Pany wrote:
>
> Hi Petar,

Hi Andre, I'm happy to help :)

> thank you very much for the explanation and the code sample.
> Filling the az_span anonymous member was the tricky part,
> I thought it would be not possible to do so, but you showed me
> the trick.

I wouldn't call it a trick, I was using standard struct literal initialization (the very syntax that DIP1031 proposes to deprecate).

For example:

struct Inner { int x, y; }
struct Outer { Inner inner; }

// You can initialize Outer in various ways:

// 1)
auto o1 = Outer(Inner(1, 2));

// 2)
Outer o2 = { inner: Inner(1, 2) };

// 3)
Outer o3 = { Inner(1, 2) };

// 4)
Outer o4 = { inner: { x: 1, y: 2} };

// 5)
Outer o5 = { { x: 1, y: 2} };

// 6)
Outer o6;
o6.inner.x = 1;
o6.inner.y = 1;

For POD (plain old data) struct like that, all six variants are equivalent (of course there more possible variations).

Since there's no `private` protection modifier in C, the only thing C library authors can do is make it inconvenient to access struct fields (by prefixing them with underscores), but they can't really prevent it.

For example, without this syntax, in pure C you can initialize a span like this:
char my_string[] = "Hey";
az_span span;
span._internal.ptr = my_string;
span._internal.length = sizeof(my_string) - 1;
span._internal.capacity = sizeof(my_string) - 1;

And with almost the same syntax you can do this in D:

string my_string = "Hey";
az_span span;
span._internal.ptr = cast(ubyte*)my_string.ptr; // note: I think this should be safe, because of [1]
span._internal.length = my_string.length;
span._internal.capacity = my_string.length;

It's just that that author wanted to prevent accidental bugs by pushing you to use the inline helper functions or macros (which are technically not needed).

[1]: https://github.com/Azure/azure-sdk-for-c/blob/25f8a0228e5f250c02e389f19d88c064c93959c1/sdk/core/core/inc/az_span.h#L22


> I will do it like you have proposed but had also already created
> a ticket for the Azure SDK developer:
> https://github.com/Azure/azure-sdk-for-c/issues/359
> There should be a more convenient way to fill a az_span structure.

To be honest, I don't think the authors will agree to change this, as putting inline functions in headers files is a pretty common practice in both C and C++.
There are two benefits to that:
1) Potentially better performance, because the code is easier to inline
2) It's possible to provide header-only libraries (not the case here), that don't require build steps.

> For reference, here is my dockerfile which does the DPP call and linking:

Cool, I'll check it later!

> ``` dockerfile
> FROM dlang2/ldc-ubuntu:1.20.0 as ldc
>
> RUN apt-get install -y git libssl-dev uuid-dev libcurl4-openssl-dev curl
>
> RUN curl -OL https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh \
>     && mkdir /opt/cmake \
>     && sh /cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake --skip-license \
>     && ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake
>
> RUN git clone https://github.com/Azure/azure-sdk-for-c.git \
>     && cd azure-sdk-for-c \
>     && git submodule update --init --recursive
>
> RUN cd azure-sdk-for-c \
>     && mkdir build \
>     && cd build \
>     && cmake ../ \
>     && make
>
> RUN apt-get install -y clang-9 libclang-9-dev
> RUN ln -s /usr/bin/clang-9 /usr/bin/clang
> COPY az_storage_blobs.dpp /tmp/
>
> RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- --help
>
> RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- /tmp/az_storage_blobs.dpp \
>     --include-path /azure-sdk-for-c/sdk/core/core/inc \
>     --include-path /azure-sdk-for-c/sdk/core/core/internal \
>     --include-path /azure-sdk-for-c/sdk/storage/blobs/inc \
>     --include-path /azure-sdk-for-c/sdk/transport_policies/curl/inc \
>     --preprocess-only
>
> ADD blobs_client_example.d /tmp/blobs_client_example.d
> RUN  ldc2 /tmp/blobs_client_example.d /tmp/az_storage_blobs.d \
>     /azure-sdk-for-c/build/sdk/core/core/libaz_core.a \
>     /azure-sdk-for-c/build/sdk/storage/blobs/libaz_storage_blobs.a \
>     /azure-sdk-for-c/build/sdk/transport_policies/curl/libaz_curl.a \
>     -of=/tmp/app
> ```
>
> Kind regards
> André

Cheers,
Petar