I've been porting dmd to Android/x86 recently and I've been thinking about whether there's an easier way to handle such porting in the future.  The vast majority of my druntime patches simply find the equivalent function or macro in the Android C library, bionic.  This is just a lot of name and parameter type checking most of the time: couldn't this be automated for new platforms with libclang?

Write a D utility that finds all the platform blocks in druntime and calls libclang to parse the headers and translate the results into D, like htod does, then inserts the new version blocks into druntime, which you can then quickly manually check.  Perhaps this can be built on top of dstep, which I just found while googling about htod for this post: :)

https://github.com/jacob-carlborg/dstep

Obviously no utility will catch everything, for example, it's not going to know how to replace missing functions.  I'm not sure if libclang provides for a way to handle architecture-specific declarations either.  But it will save a bunch of time for druntime porters, and even catch bugs in the existing manually ported platforms for druntime.  Of course, it will only work on existing clang platforms, but I doubt we're going to beat clang to a platform. ;)

It could also provide a way to separate out glibc from linux, as we've been talking about on my Android stdc pull request.  After checking the relevant C header against druntime, this utility could query the local package manager, pacman in the case of Arch linux, to see what package owned that C header and then separate glibc from linux version blocks by using that info.

I don't have any experience with libclang, so I thought I'd run the idea by you all first.  Let me know if you see any problems with this idea.

On 2014-01-07 07:02, Joakim wrote:
> I've been porting dmd to Android/x86 recently and I've been thinking
> about whether there's an easier way to handle such porting in the
> future.  The vast majority of my druntime patches simply find the
> equivalent function or macro in the Android C library, bionic.  This is
> just a lot of name and parameter type checking most of the time:
> couldn't this be automated for new platforms with libclang?
>
> Write a D utility that finds all the platform blocks in druntime and
> calls libclang to parse the headers and translate the results into D,
> like htod does, then inserts the new version blocks into druntime, which
> you can then quickly manually check.  Perhaps this can be built on top
> of dstep, which I just found while googling about htod for this post: :)
>
> https://github.com/jacob-carlborg/dstep

That could probably be built on top of DStep or a modified version of DStep.

> Obviously no utility will catch everything, for example, it's not going
> to know how to replace missing functions.

For what case one could add a static assert.

> I'm not sure if libclang
> provides for a way to handle architecture-specific declarations either.

Unfortunately it doesn't. I need some way to deal with the preprocessor in DStep as well, also the Clang developers have showed interested in be able to deal with the preprocesssor from libclang.

Clang can of course deal with this. We probably need to expose an libclang API for PPCallbacks: http://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html

> I don't have any experience with libclang, so I thought I'd run the idea
> by you all first.  Let me know if you see any problems with this idea.

Any help on improving DStep is much appreciated. I can help you out if you have an questions about DStep and probably on libclang as well.

-- 
/Jacob Carlborg

On Tuesday, 7 January 2014 at 07:36:36 UTC, Jacob Carlborg wrote:
>> I'm not sure if libclang
>> provides for a way to handle architecture-specific declarations either.
>
> Unfortunately it doesn't. I need some way to deal with the preprocessor in DStep as well, also the Clang developers have showed interested in be able to deal with the preprocesssor from libclang.
>
> Clang can of course deal with this. We probably need to expose an libclang API for PPCallbacks: http://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html

That's too bad, as the preprocessor is important in the headers.  Perhaps this can be finessed by passing various architectures or other preprocessor defines to the input of clang/libclang and then comparing the output of the various preprocessed files?  For the limited uses of the preprocessor that are translated for druntime, maybe that will be enough.

>> I don't have any experience with libclang, so I thought I'd run the idea
>> by you all first.  Let me know if you see any problems with this idea.
>
> Any help on improving DStep is much appreciated. I can help you out if you have an questions about DStep and probably on libclang as well.

At this point, I don't need such a tool, as I'm done doing most of it manually for Android/x86.  Maybe later, when we split linux from glibc and Android into linux and bionic or when I add Android/ARM support, we can look into building such a tool.

On 2014-01-07 12:17, Joakim wrote:

> That's too bad, as the preprocessor is important in the headers. Perhaps
> this can be finessed by passing various architectures or other
> preprocessor defines to the input of clang/libclang and then comparing
> the output of the various preprocessed files?  For the limited uses of
> the preprocessor that are translated for druntime, maybe that will be
> enough.

Seems complicated. I need to be able to support the preprocessor anyway in DStep. The correct solution is to extend the libclang API for this based on PPCallbacks.

-- 
/Jacob Carlborg

On Tuesday, 7 January 2014 at 12:24:25 UTC, Jacob Carlborg wrote:
> On 2014-01-07 12:17, Joakim wrote:
>
>> That's too bad, as the preprocessor is important in the headers. Perhaps
>> this can be finessed by passing various architectures or other
>> preprocessor defines to the input of clang/libclang and then comparing
>> the output of the various preprocessed files?  For the limited uses of
>> the preprocessor that are translated for druntime, maybe that will be
>> enough.
>
> Seems complicated. I need to be able to support the preprocessor anyway in DStep. The correct solution is to extend the libclang API for this based on PPCallbacks.

The "correct solution" isn't always the quick solution. ;) Anyway, thanks for building such a nice-sounding tool in DStep and for responding.  I'll let you know when I start playing with DStep.

On 2014-01-07 16:46, Joakim wrote:

> The "correct solution" isn't always the quick solution. ;) Anyway,
> thanks for building such a nice-sounding tool in DStep and for
> responding.  I'll let you know when I start playing with DStep.

Thanks.

There's another problem with DStep. Many types and functions, especially when we're talking about the system API's, are declared in internal headers and then a typedef, or similar, is used to include it in standard headers. DStep will not resolve the typedef and it will follow the same file structure as the headers do. The reason for not resolving typedefs is they're most likely hiding platform specifics in a cross-platform API.

-- 
/Jacob Carlborg