Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.  I am working in  LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success.

I have a module

// mpi.dpp

module mpi;

#include <mpi.h>

/**
 * Convert D's string[] args to a correct C char** argv.
 */
char** toArgv(string[] args)
{
    auto argv = new char*[args.length + 1];
    foreach(i, arg; args)
    {
        argv[i] = (arg.dup ~ '\0').ptr;
    }
    argv[args.length] = null;
    return argv.ptr;
}

enum MPI_DUMMY = 999;


a main module

// hello_d.d
// hacked out of the hello_d example from the OpenMPI project.

import std.stdio;
import std.string;
import std.array;
import std.algorithm;
import mpi;

int main(string[] args)
{
    int argc = cast(int)args.length;
    auto argv = args.toArgv();

    int rank;
    int size;
    writeln("MPI_VERSION=", MPI_VERSION);

    MPI_Init(&argc, &argv);
    //MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    //MPI_Comm_size(MPI_COMM_WORLD, &size);
    writefln("Hello, world, I am %d of %d", rank+1, size);
    writeln("MPI_DUMMY=", MPI_DUMMY);
    //MPI_Barrier(MPI_COMM_WORLD);
    MPI_Finalize();

    return 0;
}

and a makefile

# makefile for hello_d, playing with dpp and MPI

hello_d : hello_d.d mpi.d
	dmd -ofhello_d hello_d.d mpi.d -L-lmpi

mpi.d : mpi.dpp
	dub run dpp -- --include-path=/usr/include/openmpi \
		--preprocess-only mpi.dpp

The dpp program successfully generates an interface module mpi.d and the hello_d executable is built and it runs.  However, not all of the needed elements for a real MPI program are present.  For example, in the main module above, MPI_COMM_WORLD is not available so the lines calling MPI_Comm_rank, MPI_Comm_size and MPI_Barrier are commented out.

It may be that I am not using dpp correctly, however, running dpp with the -- option gives a lot output that includes

Cursor(MacroDefinition, "MPI_COMM_WORLD", Type(Invalid, "")) CAN @ SourceRange("/usr/include/openmpi/mpi.h", 1034:9, 1034:78)

so I am guessing that dpp does not know how to compute its type and value.

Thoughts or suggestions on how to proceed?

On 13/03/2020 9:08 PM, Peter Jacobs wrote:
> /**
>   * Convert D's string[] args to a correct C char** argv.
>   */
> char** toArgv(string[] args)
> {
>      auto argv = new char*[args.length + 1];
>      foreach(i, arg; args)
>      {
>          argv[i] = (arg.dup ~ '\0').ptr;
>      }
>      argv[args.length] = null;
>      return argv.ptr;
> }

https://dlang.org/phobos/core_runtime.html#.Runtime.cArgs

On Friday, 13 March 2020 at 08:13:44 UTC, rikki cattermole wrote:
>
> https://dlang.org/phobos/core_runtime.html#.Runtime.cArgs

Oh, that's neat.  There's so much that I don't know...

On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
> Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.  I am working in  LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success.
>
> [...]

DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files.
In both cases you have to reimplement the functions in D.

Kind regards
Andre

On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
>
> DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files.
> In both cases you have to reimplement the functions in D.
>

With my crappy binding generator it might be possible to generate some of this, however it will likely just suck given its current shape. Especially compared to dpp. 100% natural pain & misery, enjoy.

It is still might be somewhat useful to run it and gather the inlines. Maybe.

https://github.com/Superbelko/ohmygentool

On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
> On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
>> Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.  I am working in  LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success.
>>
>> [...]
>
> DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files.
> In both cases you have to reimplement the functions in D.
>
> Kind regards
> Andre

Some of that can be done by dstep:
https://github.com/jacob-carlborg/dstep

On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
> On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
>> Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.  I am working in  LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success.
>>
>> [...]
>
> DPP cannot translate macros.

It can translate some macros, but not all.

> Another problem is inlined functions.

Yes. Technically I could add support for them by adding translation functions for all possible C AST nodes (I shudder to even consider C++ here), but it'd be quite a bit of work.

> While DPP generates their headers, their implementations are missing in the binary files.
> In both cases you have to reimplement the functions in D.

Not quite - you can force a C compiler to emit code for them and link to it, using the dpp-translated declaration during compilation. It's how D supports calling C++ template functions now.

On Friday, 13 March 2020 at 13:00:32 UTC, bachmeier wrote:
> On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
>> On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
>>> Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.
>
> Some of that can be done by dstep:
> https://github.com/jacob-carlborg/dstep

Tried the most recent dstep but was unsuccessful...

$ ../dstep/bin/dstep /usr/include/mpi/mpi.h -o mpi.d
/usr/include/mpi/mpi.h:351:8: warning: a type renamed to 'ompi_status_public_t_' due to the collision with the symbol declared in /usr/include/mpi/mpi.h:364:37
dstep: an unknown error occurred: core.exception.AssertError@dstep/translator/Type.d(32): Assertion failure
----------------
??:? _d_assertp [0x5625185d8659]
dstep/translator/Type.d:32 dstep.translator.Output.SourceNode dstep.translator.Type.translateType(dstep.translator.Context.Context, clang.Cursor.Cursor, clang.Type.Type, bool, bool) [0x5625185c1941]

--- snip ---

clang/Visitor.d:66 extern (C) clang.c.Index.CXChildVisitResult clang.Visitor.Visitor.visitorFunction(clang.c.Index.CXCursor, clang.c.Index.CXCursor, void*) [0x562518562feb]
??:? [0x7f7d196f7bff]

On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
> Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.  I am working in  LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success.
>
> I have a module
>
> // mpi.dpp
>
> module mpi;
>
> #include <mpi.h>
>
> /**
>  * Convert D's string[] args to a correct C char** argv.
>  */
> char** toArgv(string[] args)
> {
>     auto argv = new char*[args.length + 1];
>     foreach(i, arg; args)
>     {
>         argv[i] = (arg.dup ~ '\0').ptr;
>     }
>     argv[args.length] = null;
>     return argv.ptr;
> }
>
> enum MPI_DUMMY = 999;
>
>
> a main module
>
> // hello_d.d
> // hacked out of the hello_d example from the OpenMPI project.
>
> import std.stdio;
> import std.string;
> import std.array;
> import std.algorithm;
> import mpi;
>
> int main(string[] args)
> {
>     int argc = cast(int)args.length;
>     auto argv = args.toArgv();
>
>     int rank;
>     int size;
>     writeln("MPI_VERSION=", MPI_VERSION);
>
>     MPI_Init(&argc, &argv);
>     //MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>     //MPI_Comm_size(MPI_COMM_WORLD, &size);
>     writefln("Hello, world, I am %d of %d", rank+1, size);
>     writeln("MPI_DUMMY=", MPI_DUMMY);
>     //MPI_Barrier(MPI_COMM_WORLD);
>     MPI_Finalize();
>
>     return 0;
> }
>
> and a makefile
>
> # makefile for hello_d, playing with dpp and MPI
>
> hello_d : hello_d.d mpi.d
> 	dmd -ofhello_d hello_d.d mpi.d -L-lmpi
>
> mpi.d : mpi.dpp
> 	dub run dpp -- --include-path=/usr/include/openmpi \
> 		--preprocess-only mpi.dpp
>
> The dpp program successfully generates an interface module mpi.d and the hello_d executable is built and it runs.  However, not all of the needed elements for a real MPI program are present.  For example, in the main module above, MPI_COMM_WORLD is not available so the lines calling MPI_Comm_rank, MPI_Comm_size and MPI_Barrier are commented out.
>
> It may be that I am not using dpp correctly, however, running dpp with the -- option gives a lot output that includes
>
> Cursor(MacroDefinition, "MPI_COMM_WORLD", Type(Invalid, "")) CAN @ SourceRange("/usr/include/openmpi/mpi.h", 1034:9, 1034:78)
>
> so I am guessing that dpp does not know how to compute its type and value.
>
> Thoughts or suggestions on how to proceed?

Have you seen this project?

https://github.com/1100110/OpenMPI

As I think about this, OpenMPI is probably one of the most difficult libraries to deal with. It's such a complicated beast to build that you're supposed to use their wrapper compiler.

On Monday, 16 March 2020 at 01:12:42 UTC, bachmeier wrote:
> On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
>> Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.
>> Thoughts or suggestions on how to proceed?
>

--- snip ---

> Have you seen this project?
>
> https://github.com/1100110/OpenMPI
>
> As I think about this, OpenMPI is probably one of the most difficult libraries to deal with. It's such a complicated beast to build that you're supposed to use their wrapper compiler.

Yes, indeed, we use that OpenMPI wrapper, actually the version at https://github.com/DlangScience/OpenMPI which has been maintained by John Colvin. A big thanks to both Jude Young and John Colvin.  We have run our CFD flow solver across more than 1000 cores using MPI http://cfcfd.mechmining.uq.edu.au/eilmer/pdfs/cfh-seminar-oct-2019.pdf

What prompted my experiments with dpp is that I have been getting reports that our code is not building with very recent editions of openmpi, presumably because the OpenMPI wrapper-generator code is surprised by the new header file content, and that I have seen dpp suggested by Atila Neves a couple of times.