Hi all,

My name is Cristi and in the next few months I will be working on the "DPP with Linux kernel headers" project for this year's edition of SAoC. This thread will be used to post further updates (though anyone is encouraged to make suggestions, ask questions and pinpoint mistakes).

DPP is a tool used to directly include C/C++ headers in D files; it is basically a D compiler wrapper which permits D files with #include directives to be compiled.
Since D doesn't have a preprocessor, running DPP on a .dpp file will create a valid .d file that can be compiled (e.g. with dmd). A .dpp file can include one or more C/C++ headers, each possibly containing C/C++ macros, and function and structure declarations. The included headers are parsed using libclang.

At the moment, DPP doesn't work when the .dpp file includes linux header files. As a consequence, writing a linux driver in D (e.g. the case of Alex Militaru's driver), implies manually translating all used C headers to .di interfaces (redeclaring all structures and functions to a D-compilable format). The project aims to solve this issue and enable DPP to work with all linux headers.

Milestones for this project:

Milestone 1: Investigate and narrow down the issues that dpp has with the compilation of linux kernel header files.
Weeks 1-2: Get accustomed with the infrastructure: build the kernel with clang, clone dpp, understand the internal structure of both dpp and the kernel.
Weeks 3-4: Compile a .c file with clang that includes a random linux kernel header, try to reproduce with dpp. There are a series of issues that need to be addressed:
       -> how to pass clang command line options to dpp?
       -> what command line options are needed?
       -> how to specify what version of libclang should be used?
       -> are the object files compatible?

Milestone 2: Fix all issues encountered at milestone 1 so that Alex Militaru's driver can be integrated with the linux kernel using dpp.
Unfortunately, we cannot detail too much on this step as it really depends on what we will discover during milestone 1. We can provide more information about this at the end of milestone 1. If things go smoothly and we integrate Alex Militaru's driver very fast, then the next step will
be to create a testing infrastructure that makes sure that dpp works well with **all** the headers in the kernel.

Milestone 3 + 4:  Work on integrating dpp with C++.

From now on, I will be posting updates weekly (or every two weeks) regarding the progress done on this project.

Quick note: I'm currently in the last two weeks of an internship, but I will do my best to finish all that I've set out to do.

Many thanks,
Cristi

On 2019-09-16 22:51, Cristian Becerescu wrote:

>         -> how to pass clang command line options to dpp?

DStep [1], which is very similar to dpp and uses libclang, will forward any unrecognized arguments to libclang. If there are any conflicts, it's always possible to pass "--" to separate Clang arguments from DStep arguments. It's common practice and is supported by std.getopts, IIRC.

>         -> how to specify what version of libclang should be used?

I would assume that dpp doesn't use `dlopen` and just uses whatever version of libclang is installed. DStep supports static linking of libclang and the pre-compiled release binaries are statically linked with libclang to minimize the risk that an unsupported version of libclang is used.

[1] https://github.com/jacob-carlborg/dstep

-- 
/Jacob Carlborg

On Monday, 16 September 2019 at 20:51:08 UTC, Cristian Becerescu wrote:
> Hi all,
>
> My name is Cristi and in the next few months I will be working on the "DPP with Linux kernel headers" project for this year's edition of SAoC. This thread will be used to post further updates (though anyone is encouraged to make suggestions, ask questions and pinpoint mistakes).
>
> [...]

You can pass command-line options directly to dpp with --clang-option.

Sorry for not updating this thread in a while.
I've managed to do some progress on this project, but from now on I'll have more time, considering I finished my internship.

Here are some updates and things I found out during those weeks:

1. Yes, as Atila said, I can pass flags to clang through dpp by using --clang-option. The only inconvenience with this is that every clang flag needs to be preceeded by a --clang-option flag (this becomes unmanageable when dealing with tens of compiler flags, e.g. for compiling kernel headers).

This is not a problem which affects the capabilities of d++, but it impacts the ease of use and increases the chances of mistyping, forgetting to add the flag or adding it in a wrong place.

Solution #1 would be defining arraySep="," so that the getOpt() function (used for parsing the command line) could accept multiple flags at once for clang-options: --clang-options=-D__KERNEL__,-Werror,-Wextra).

Solution #2 could be providing a string containing all clang options: --clang-options "-D__KERNEL__ -Werror -Wextra", which would be more convenient when copy-pasting flags from the kernel build files.

2. The LLVM compiler infrastructure doesn't yet support the 'asm goto' gcc extension.

I encountered this problem while trying to compile a simple, empty main C program which #included linux/namei.h.
If the kernel is built with CONFIG_JUMP_LABEL and CC_HAVE_ASM_GOTO set, using clang will not work (for details, see asm/compiler.h, asm/compiler_types.h, asm/compiler-gcc.h, asm/compiler-clang.h). There is a macro (asm_volatile_goto(x)) used in asm/jump_label.h which is declared in asm/compiler-gcc.h, but not in asm/compiler-clang.h (asm/jump_label.h includes one of the two, but always uses the macro, even though in one of the included files the macro is not defined). Undefining CC_HAVE_ASM_GOTO (or just not defining it with -D...) when compiling the .c file worked.

The motivation of debugging this was to first make sure that I can compile a .c file (containing a kernel header) using clang, so that then, when I compile a .dpp file using d++ (and clang internally) and get errors, I know that there are problems withing dpp and not clang.

3. After making sure clang worked with the above .c program, I tried compiling the .dpp one with the same clang options preceded by --clang-option. This currently does not work, as clang reports some "undeclared identifier" and other syntax errors which I'll have to dig into deeper to understand whether it's a problem with the way the flags are passed to clang or something else.

It's time for a new update, so here we go. Sorry for those long posts :)

This past week I've dived deeper into the 3rd problem mentioned in the last update.
There were multiple problems when trying to generate and compile a D program from this DPP:

// foo.dpp
#include <linux/namei.h>

void main()
{
}

Problem #1 - Clang errors
Even though compiling a C program which included the same kernel header with clang worked, compiling it through dpp didn't. I've managed to find out that this happens because dpp appends some include directories by default, the problem being related to '/usr/include/' in particular (which is set through the function call systemPaths() from the D libclang [1]). Clang complains about undeclared identifiers and expected closed parenthesis, but I still have to investigate why including that directory to the include directories messes things up.

-------------------------------------------------

Problem #2 - Multi-line macro definition w/ casts
Consider this code from kernel.h:

#define u64_to_user_ptr(x) (		\
{					\
	typecheck(u64, x);		\
	(void __user *)(uintptr_t)x;	\
}					\					
)

In dpp, when translating this into D code, we also check for pairs of open and closed parenthesis [2]. When finding a '(', we increment the index of the tokens array until we find a matching ')'. If the C code is valid (and in the above example it is), this should work well, but it doesn't, resulting in a fatal error: range violation. The reason is, as seen in [2], we only check for tokens with the ')' spelling, when, in reality, the last parenthesis of the macro is not spelled ')', but '\\\n)' ('\' character, followed by newline, followed by the actual parenthesis).

-------------------------------------------------

Problem #3 - Aggregates with name being a D keyword
I'll give you an example:

// test.h
struct module;

void f(struct module *);

struct module {
	int a;
	int b;
};

Generating a D file from a DPP one which includes the above header will look like this:

// test.d, generated from test.dpp through the last version of d++ from github
// ...
extern(C)
{
	void f(module*) @nogc nothrow;
	struct module__
	{
		int a;
		int b;
	}
}

struct module;

void main() {}
// ...

Clearly there are multiple wrong things here:
- the module struct should be named module_ and not module__ (this is what dpp should do internally)
- even though the struct is renamed, the parameter types are not
- we are declaring the same structure again (with the original C spelling) outside of the extern(C) block because dpp thinks module was an undeclared structure
- compilation of this D program clearly doesn't work

The reasons for those bugs are a bit subtler, so I'm not going over them as it would make this post quite big.

-------------------------------------------------

I have implemented working solutions for all the above problems. They pass all the unit tests and I can also successfully generate an executable file from a .dpp which contains the linux/namei.h kernel header. I just have to clean some things and start making pull requests and maybe get some feedback.
Problem #1 is "solved" by not including that directory in the paths (just a workaround for the moment), but as I will investigate this further, I will try to see what the underlying problem really is (probably some collisions with other files).

Going from here, I will investigate if my changes impact other non-unit-tested C cases. Also, I will try running dpp with other kernel and non-kernel headers as well, making sure there are no other bugs or untreated edge cases.

Cristi


[1] https://code.dlang.org/packages/libclang
[2] https://github.com/atilaneves/dpp/blob/master/source/dpp/translation/macro_.d#L326

On Thursday, 10 October 2019 at 20:28:20 UTC, Cristian Becerescu wrote:
> It's time for a new update, so here we go. Sorry for those long posts :)
>
> [...]

Could you please file issues on github so I can fix them? Thanks.

On Friday, 11 October 2019 at 13:11:48 UTC, Atila Neves wrote:
> On Thursday, 10 October 2019 at 20:28:20 UTC, Cristian Becerescu wrote:
>> It's time for a new update, so here we go. Sorry for those long posts :)
>>
>> [...]
>
> Could you please file issues on github so I can fix them? Thanks.

It looks like Cristian has already solved these issues. He will probably make some PRs next week.

Update for week 1 of Milestone 2.

I've implemented solutions for the issues discovered in the previous 2 weeks and made pull requests as follows:

- Allow multiple clang options to be specified more easily & add option to avoid including the system paths by default
https://github.com/atilaneves/dpp/pull/185

- Fix a bug related to multi-line macro definitions
https://github.com/atilaneves/dpp/pull/186

- Fix the renaming of D keywords to be consistent and correct
https://github.com/atilaneves/dpp/pull/187

Update for week 2 of Milestone 2

- PRs from the previous week were successfully merged

- Tested running dpp to translate linux/virtio.h

In this phase, I've begun testing dpp with every linux header used by Alex Militaru's driver, starting with linux/virtio.h.

There are multiple cases where a function is declared through a macro, where the function return type is specified using __typeof(X)__ (e.g. __typeof(X)__ functionName()). The D translation would be something like:

typeof(X) functionName();

In C, X could be a variable of a certain type, the type itself (e.g. int), or an alias of a type.
In D, it is not valid code if X is either a certain type, or an alias of a type.

E.g.:
alias u32 = uint;

typeof(u32) f1(); // err
typeof(int) f2(); // err

Another issue is related to translating nested (on multiple levels) anonymous unions and structs (specifically in linux/slab.h, line 596), but I still have to wrap my head around this (because the generated code is a bit complicated).

An inconvenience is that on my machine, running dpp with virtio.h takes about 15 minutes, which slows down the debugging process with this specific header.

On Monday, 28 October 2019 at 11:14:24 UTC, Cristian Becerescu wrote:
> Update for week 2 of Milestone 2
>
> - PRs from the previous week were successfully merged
>
> [...]

Maybe you can try:

u32 i=0;
typeof(u32.init) f1(); // err
typeof(int.init) f2(); // err
typeof(i.init) f2(); // err