Hi, I would like to announce llvm-d, which provides LLVM bindings for D.

It loads LLVM from a dynamic library (so/dylib/dll) and has support
for LLVM versions 3.1, 3.2 and 3.3 (current svn trunk).
It has as of now been tested on 64bit versions of Archlinux, OS X Mountain Lion and Windows 7.

The support for multiple versions is achieved via conditional compilation (As the LLVM C API sometimes has changes such as insertion of new enum items
in the middle of an enum, changing the values for all succeeding enum items), specifically the version system: LLVM version X.Y translates to the version identifier "LLVM_X_Y" (without the quotes).

More information (and an example) can be found at llvm-d's location:

https://github.com/Calrama/llvm-d

and also has a DUB registry entry at

http://registry.vibed.org/packages/llvm-d

Moritz Maxeiner wrote:
> Hi, I would like to announce llvm-d, which provides LLVM bindings for D.
> 
> It loads LLVM from a dynamic library (so/dylib/dll) and has support
> for LLVM versions 3.1, 3.2 and 3.3 (current svn trunk).
> It has as of now been tested on 64bit versions of Archlinux, OS X
> Mountain Lion and Windows 7.
> 
> The support for multiple versions is achieved via conditional
> compilation (As the LLVM C API sometimes has changes such as
> insertion of new enum items
> in the middle of an enum, changing the values for all succeeding
> enum items), specifically the version system: LLVM version X.Y
> translates to the version identifier "LLVM_X_Y" (without the
> quotes).
> 
> More information (and an example) can be found at llvm-d's location:
> 
> https://github.com/Calrama/llvm-d
> 
> and also has a DUB registry entry at
> 
> http://registry.vibed.org/packages/llvm-d

Nice.
Can you make it compatible with Deimos
https://github.com/D-Programming-Deimos/?

Jens

On Friday, 15 March 2013 at 21:18:09 UTC, Jens Mueller wrote:
> Nice.
> Can you make it compatible with Deimos
> https://github.com/D-Programming-Deimos/?
>
> Jens

If by compatible you mean "can be used with":

I don't see anything that would prevent you from using llvm-d together
with deimos bindings for other libraries.

If by compatible you mean "can be integrated into":

Afaik Deimos projects are linked with the libraries, instead of loading
them at runtime. This is something I do not want because of the coff/omf
hassle with optlink (on win32), which I can avoid by sidestepping the linker.

The next problem would be that afaik deimos is for pure C bindings only, but as mentioned under "Planned Features" on the github page I do plan on more or less recreating LLVM's OOP structure (as much as the C API exposes, anyway) in D around the C API as close to the original as sensible and feasible. Keeping two seperate projects for that seemed inefficient to me, even more so when one of them would only change for bug fixes (if any) and for new LLVM versions.

Another pro for using runtime loading is that I can move the names of the libraries into llvm-d (which is what I did), so someone using llvm-d does not have to worry about library names, prefixes and suffixes. The most they have to do it to give -ldl to the linker on posix.

These are the main reasons as to why I decided against making another deimos project in the first place and I can't think of a compelling reason for deviating from that decision right now, sorry. Correct me if I'm wrong, though, there's much about D I don't know yet.

If, however, there are enough people that *really* want a deimos version I could fork the project, strip out the shared lib stuff and change a few calls in the code generation to create extern functions instead of extern function pointers (The process is automated via CTFE, a mixin version of the "map" function and associative array enums) and that would be it, although I don't really see any real benefit to that (other than it being in deimos, which I don't think is a key factor now that we have dub with its package registry around).

Moritz Maxeiner wrote:
> On Friday, 15 March 2013 at 21:18:09 UTC, Jens Mueller wrote:
> >Nice.
> >Can you make it compatible with Deimos
> >https://github.com/D-Programming-Deimos/?
> >
> >Jens
> 
> If by compatible you mean "can be used with":
> 
> I don't see anything that would prevent you from using llvm-d
> together
> with deimos bindings for other libraries.
> 
> If by compatible you mean "can be integrated into":
> 
> Afaik Deimos projects are linked with the libraries, instead of
> loading
> them at runtime. This is something I do not want because of the
> coff/omf
> hassle with optlink (on win32), which I can avoid by sidestepping
> the linker.

Deimos just provides a plain D interface for C libraries.
What do you mean?
You can do static or dynamic linking.
With dynamic linking it will be loaded at startup time by the loader.
But you can also load it at runtime using some system API which I assume
you do. I started ddl (http://jkm.github.com/ddl/ddl.html) which does
work with Deimos to some extent. A limiting factor is compilation speed.

> The next problem would be that afaik deimos is for pure C bindings only, but as mentioned under "Planned Features" on the github page I do plan on more or less recreating LLVM's OOP structure (as much as the C API exposes, anyway) in D around the C API as close to the original as sensible and feasible. Keeping two seperate projects for that seemed inefficient to me, even more so when one of them would only change for bug fixes (if any) and for new LLVM versions.

Even though I understand your arguments. Technically, they are different projects. As you say built on LLVM's C API. One could use git submodules to ease building the higher level API.

> Another pro for using runtime loading is that I can move the names of the libraries into llvm-d (which is what I did), so someone using llvm-d does not have to worry about library names, prefixes and suffixes. The most they have to do it to give -ldl to the linker on posix.

Very sensible. Yet this is something you want in general. ddl is able to do this.

> These are the main reasons as to why I decided against making another deimos project in the first place and I can't think of a compelling reason for deviating from that decision right now, sorry. Correct me if I'm wrong, though, there's much about D I don't know yet.

In sum you want to load libraries at runtime and have a higher level API. Both are fine. But I question that it is a good decision to do the loading at runtime for each C library.

> If, however, there are enough people that *really* want a deimos version I could fork the project, strip out the shared lib stuff and change a few calls in the code generation to create extern functions instead of extern function pointers (The process is automated via CTFE, a mixin version of the "map" function and associative array enums) and that would be it, although I don't really see any real benefit to that (other than it being in deimos, which I don't think is a key factor now that we have dub with its package registry around).

I would use Deimos bindings. I even started one myself.
https://github.com/jkm/deimos-llvm/
But I cannot devote much time on it.

Jens

On Sunday, 17 March 2013 at 20:05:09 UTC, Jens Mueller wrote:
> Deimos just provides a plain D interface for C libraries.
> What do you mean?
> You can do static or dynamic linking.
> With dynamic linking it will be loaded at startup time by the loader.
> But you can also load it at runtime using some system API which I assume
> you do. I started ddl (http://jkm.github.com/ddl/ddl.html) which does
> work with Deimos to some extent. A limiting factor is compilation speed.

Technically deimos provides extern(C) functions, which by themselves have - as  you said - to be linked statically/dynamically and cannot be loaded at runtime.
Normally to load at runtime you need a function pointer to give the OS functions to load from the shared lib and afaict you're using ctfe to extract the functions from deimos and create those functions pointers from them to circumvent that problem - kudos by the way, never even thought about that possibility.
Now to the real problem: When you're creating a shared lib on windows (32bit) it's going to be COFF, normally. But optlink can't link with that, only with OMF - which is one of the reason, I believe, why support for the MSLinker was added to windows 64bit versions of dmd. You can use import libs to make dmd on win32 work with COFF shared libs, but again, that is extra work on the user of llvm-d, which I do not want. The solution for me was simply to cut out the middle man reponsible for that hassle (optlink) from the process.

On another note:
- Afaict ddl doesn't have OS X support, so I can't use it as a replacement for what I already have. If you want to add that to ddl you could just look at llvm/util/shlib.d in llvm-d, it's done there, missing support for one of the three majors is a big no-no for me.
- Another problem would be that I want to keep things compatible with dub, which would become very difficult seing as ddl doesn't have support for dub at present.
- Moreover, ddl doesn't seem to have support for loading from a specific directory (such as "lib"), or at least I can't find it
- There are functions in the LLVM C API, which are not available in every LLVM shared lib: The target specific ones. That means that - depending on which targets where selected when compiling th shared lib - these functions may or may not be in the shared lib and trying to load them from the shared lib MUST NOT throw an exception. If loading fails it simply means the shared lib doesn't have support for that specific target; ddl however, always throws an exception when it can't load a symbol.

Only with those issues resolved in ddl would it be even feasible for me to do a split into say llvm-d + (deimos)llvm and have llvm-d depend on ddl.
And even then llvm-d would depend on 2 projects (ddl and (deimos)llvm) instead of 0 like now and more dependencies for very little to no functionality increase is a very big argument against doing so for me.

> Even though I understand your arguments. Technically, they are different
> projects. As you say built on LLVM's C API. One could use git submodules
> to ease building the higher level API.

By that same argument, however, you could also claim that the D runtime core should be split into more projects - one who only imports the necessary C functions from the libc, and the other who does all the D stuff.
Not the best example, I know, but the point still stands: When the D OOP API is done, the llvm-d user should have no more need to use the C functions by themselves. Also LLVM A API functions often do strange things you wouldn't expect and the documentation is sometimes just simply wrong; using the LLVM C API from D with D's GC without something in between that takes care of the GC/NonGC gap is dangerous at best, I believe.

> In sum you want to load libraries at runtime and have a higher level
> API. Both are fine. But I question that it is a good decision to do the
> loading at runtime for each C library.

Not that I advocate doing it for every C library, but if one really wants to what would be the real downside? The program startup time would suffer, of course, but with current computer architectures would it really be by an amount that actually hurts? And would the comfort of not having to worry about different library formats and linkers  on different OS's outweigh that? It's not like it is actually calculation-heavy stuff, only loading symbols. You could even parallelise it if it's more than a couple of hundred.

> I would use Deimos bindings. I even started one myself.
> https://github.com/jkm/deimos-llvm/
> But I cannot devote much time on it.

That would make it one for and one somewhat against^^
In all seriousness, though: Even if I were to split llvm-d into the two parts, I can't do it right now, as only the C API stuff exists at present, so splitting would leave me with one empty github repository right now.
What I could do, however, is copying the C parts from llvm-d instead of splitting them. If I were to do that, I'd need to know how to get it in the deimos "D programming" group, though.

To sum it up: I'd much rather make a copy of the C stuff from llvm-d, make it a deimos project, but leave llvm-d untouched then to do a split and deal with the consequences. It might seem contraproductive, but the actual effort to support two very similar projects with updates is less than to deal with problems arising from the split.

- Moritz

Moritz Maxeiner wrote:
> On Sunday, 17 March 2013 at 20:05:09 UTC, Jens Mueller wrote:
> >Deimos just provides a plain D interface for C libraries.
> >What do you mean?
> >You can do static or dynamic linking.
> >With dynamic linking it will be loaded at startup time by the
> >loader.
> >But you can also load it at runtime using some system API which I
> >assume
> >you do. I started ddl (http://jkm.github.com/ddl/ddl.html) which
> >does
> >work with Deimos to some extent. A limiting factor is compilation
> >speed.
> 
> Technically deimos provides extern(C) functions, which by themselves
> have - as  you said - to be linked statically/dynamically and cannot
> be loaded at runtime.
> Normally to load at runtime you need a function pointer to give the
> OS functions to load from the shared lib and afaict you're using
> ctfe to extract the functions from deimos and create those functions
> pointers from them to circumvent that problem - kudos by the way,
> never even thought about that possibility.
> Now to the real problem: When you're creating a shared lib on
> windows (32bit) it's going to be COFF, normally. But optlink can't
> link with that, only with OMF - which is one of the reason, I
> believe, why support for the MSLinker was added to windows 64bit
> versions of dmd. You can use import libs to make dmd on win32 work
> with COFF shared libs, but again, that is extra work on the user of
> llvm-d, which I do not want. The solution for me was simply to cut
> out the middle man reponsible for that hassle (optlink) from the
> process.

This makes perfect sense. I understand why that was your solution. But it is limiting for non-Windows users.

> On another note:
> - Afaict ddl doesn't have OS X support, so I can't use it as a
> replacement for what I already have. If you want to add that to ddl
> you could just look at llvm/util/shlib.d in llvm-d, it's done there,
> missing support for one of the three majors is a big no-no for me.

I assumed OS X is Posix compliant. I never checked that though. Thanks for the pointer.

> - Another problem would be that I want to keep things compatible with dub, which would become very difficult seing as ddl doesn't have support for dub at present.

No problem. I can add this. It is easy to do. I didn't do it because I wasn't sure which package manager will be the "official" one. This hasn't been settled as far as I know.

> - Moreover, ddl doesn't seem to have support for loading from a specific directory (such as "lib"), or at least I can't find it

I don't know how this is done on Windows. On Linux you just set the LD_LIBRARY_PATH.

> - There are functions in the LLVM C API, which are not available in every LLVM shared lib: The target specific ones. That means that - depending on which targets where selected when compiling th shared lib - these functions may or may not be in the shared lib and trying to load them from the shared lib MUST NOT throw an exception. If loading fails it simply means the shared lib doesn't have support for that specific target; ddl however, always throws an exception when it can't load a symbol.

Uhh. I didn't know. Unfortunately I have very little experience with the LLVM C API. But it is a very important use case for ddl and Deimos in general.

> Only with those issues resolved in ddl would it be even feasible for
> me to do a split into say llvm-d + (deimos)llvm and have llvm-d
> depend on ddl.
> And even then llvm-d would depend on 2 projects (ddl and
> (deimos)llvm) instead of 0 like now and more dependencies for very
> little to no functionality increase is a very big argument against
> doing so for me.

I see. It's up to you. You know my preference.

> >Even though I understand your arguments. Technically, they are
> >different
> >projects. As you say built on LLVM's C API. One could use git
> >submodules
> >to ease building the higher level API.
> 
> By that same argument, however, you could also claim that the D
> runtime core should be split into more projects - one who only
> imports the necessary C functions from the libc, and the other who
> does all the D stuff.
> Not the best example, I know, but the point still stands: When the D
> OOP API is done, the llvm-d user should have no more need to use the
> C functions by themselves. Also LLVM A API functions often do
> strange things you wouldn't expect and the documentation is
> sometimes just simply wrong; using the LLVM C API from D with D's GC
> without something in between that takes care of the GC/NonGC gap is
> dangerous at best, I believe.

Probably. There are very few users for such a use case.
But if you would use LLVM's plain C interface in D users who have used
the API in C will feel home. I mean why expose the LLVM C API as an
addition. It certainly does no harm.

> >In sum you want to load libraries at runtime and have a higher
> >level
> >API. Both are fine. But I question that it is a good decision to
> >do the
> >loading at runtime for each C library.
> 
> Not that I advocate doing it for every C library, but if one really wants to what would be the real downside? The program startup time would suffer, of course, but with current computer architectures would it really be by an amount that actually hurts? And would the comfort of not having to worry about different library formats and linkers  on different OS's outweigh that? It's not like it is actually calculation-heavy stuff, only loading symbols. You could even parallelise it if it's more than a couple of hundred.

Usually, I use dynamic linking. I believe you would do the same if optlink worked.

> >I would use Deimos bindings. I even started one myself.
> >https://github.com/jkm/deimos-llvm/
> >But I cannot devote much time on it.
> 
> That would make it one for and one somewhat against^^
> In all seriousness, though: Even if I were to split llvm-d into the
> two parts, I can't do it right now, as only the C API stuff exists
> at present, so splitting would leave me with one empty github
> repository right now.
> What I could do, however, is copying the C parts from llvm-d instead
> of splitting them. If I were to do that, I'd need to know how to get
> it in the deimos "D programming" group, though.

The process is usually pretty simple. But the documentation is lacking. There are several pieces floating around. I can try writing one document and try to push it to the official web site.

> To sum it up: I'd much rather make a copy of the C stuff from llvm-d, make it a deimos project, but leave llvm-d untouched then to do a split and deal with the consequences. It might seem contraproductive, but the actual effort to support two very similar projects with updates is less than to deal with problems arising from the split.

Do as you like it. I do not think this is a bad approach.

Jens

On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:
>
> I assumed OS X is Posix compliant. I never checked that though. Thanks
> for the pointer.

No problem, OSX seems mostly Posix compliant. The shared libraries are suffixed with ".dylib", though, instead of ".so". There may be other differences you need to deal with, but that's the only one I ran into in my minimalistic approach.

> No problem. I can add this. It is easy to do. I didn't do it because I
> wasn't sure which package manager will be the "official" one. This
> hasn't been settled as far as I know.

You're right, of course, but at present time dub has a package registry and is easy to use, so I went with that. If, however, there is going to be an official package manager at some point in the future I'll change to that, then.

>> - Moreover, ddl doesn't seem to have support for loading from a
>> specific directory (such as "lib"), or at least I can't find it
>
> I don't know how this is done on Windows. On Linux you just set the
> LD_LIBRARY_PATH.

My problem with that is that it's a) not encapsulated inside the program and b) OS dependent.

> Probably. There are very few users for such a use case.
> But if you would use LLVM's plain C interface in D users who have used
> the API in C will feel home. I mean why expose the LLVM C API as an
> addition. It certainly does no harm.

The C API will keep being exposed, of course; all I meant was that splitting them off might not be the best idea when they mostly exist only to be used by the D API and the C API should not be used unless you really want/have to.

> Usually, I use dynamic linking. I believe you would do the same if
> optlink worked.

Maybe if optlink had worked with COFF when I started using D (back in D1 times), but right now, even if optlink would start working on win32 with COFF I'd still go with runtime loading as often as possible (Not counting D libraries with D classes in them, of course, only C API libraries). Mostly because this way you can start your programs with "rdmd program.d". No need to care at all about libraries: Getting a working compilation should imho be as simple as invoking the compiler with the source file that contains your main (or the root file of the library to create). Extra argument are necessary for specific configuration, but compiling should work with the least amount of extra argument as possible.

> The process is usually pretty simple. But the documentation is lacking.
> There are several pieces floating around. I can try writing one document
> and try to push it to the official web site.

That'd be great, thank you very much.

> Do as you like it. I do not think this is a bad approach.

I'll to that, then. I'm currently working on the D API, but I'll try to get the copied deimos compatible project started in the next couple of days.

- Moritz

Moritz Maxeiner wrote:
> On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:
> >I don't know how this is done on Windows. On Linux you just set
> >the
> >LD_LIBRARY_PATH.
> 
> My problem with that is that it's a) not encapsulated inside the
> program and b) OS dependent.

Very true. Do you have a solution for this? I can add it to ddl, if that is the right place to put it.

> >Probably. There are very few users for such a use case.
> >But if you would use LLVM's plain C interface in D users who have
> >used
> >the API in C will feel home. I mean why expose the LLVM C API as
> >an
> >addition. It certainly does no harm.
> 
> The C API will keep being exposed, of course; all I meant was that splitting them off might not be the best idea when they mostly exist only to be used by the D API and the C API should not be used unless you really want/have to.

And by splitting you mean in different files, right?

> >Usually, I use dynamic linking. I believe you would do the same if optlink worked.
> 
> Maybe if optlink had worked with COFF when I started using D (back in D1 times), but right now, even if optlink would start working on win32 with COFF I'd still go with runtime loading as often as possible (Not counting D libraries with D classes in them, of course, only C API libraries). Mostly because this way you can start your programs with "rdmd program.d". No need to care at all about libraries: Getting a working compilation should imho be as simple as invoking the compiler with the source file that contains your main (or the root file of the library to create). Extra argument are necessary for specific configuration, but compiling should work with the least amount of extra argument as possible.

I see.

> >The process is usually pretty simple. But the documentation is
> >lacking.
> >There are several pieces floating around. I can try writing one
> >document
> >and try to push it to the official web site.
> 
> That'd be great, thank you very much.
> 
> >Do as you like it. I do not think this is a bad approach.
> 
> I'll to that, then. I'm currently working on the D API, but I'll try to get the copied deimos compatible project started in the next couple of days.

Sounds good. I'm going to write the documentation this evening.

Jens

On Monday, 18 March 2013 at 14:51:00 UTC, Jens Mueller wrote:
> Moritz Maxeiner wrote:
>> On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:
>> >I don't know how this is done on Windows. On Linux you just set
>> >the
>> >LD_LIBRARY_PATH.
>> 
>> My problem with that is that it's a) not encapsulated inside the
>> program and b) OS dependent.
>
> Very true. Do you have a solution for this? I can add it to ddl, if that
> is the right place to put it.

Well, it's more of a workaround than a solution:
The loading function (D) takes two arguments: The path and the file(name)
of the shared lib. If no filename is given, a ctfe generated one is used. The shared contructors calls this function with a path of "". That's about all I need for the LLVM shared lib; but if you're going to load a shared lib from a certain path which have dependencies in that certain path you'll need to additionally use SetDLLDirectoryA on windows (posix automatically looks in the directory a shared lib is loaded from afaik).
For source code you could look here:
https://github.com/Calrama/llvm-d/blob/master/llvm/c/all.d

>
>> >Probably. There are very few users for such a use case.
>> >But if you would use LLVM's plain C interface in D users who have
>> >used
>> >the API in C will feel home. I mean why expose the LLVM C API as
>> >an
>> >addition. It certainly does no harm.
>> 
>> The C API will keep being exposed, of course; all I meant was that
>> splitting them off might not be the best idea when they mostly exist
>> only to be used by the D API and the C API should not be used unless
>> you really want/have to.
>
> And by splitting you mean in different files, right?

No, the C API will rest in llvm/c/... (as it does right now) while the D API will rest in llvm/d/... (so the C API will be usable without the D API and can be compiled and linked in without it).
With "splitting" I meant putting them in another project entirely like llvm-d and (deimos)llvm.

>> >The process is usually pretty simple. But the documentation is
>> >lacking.
>> >There are several pieces floating around. I can try writing one
>> >document
>> >and try to push it to the official web site.
>> 
>> That'd be great, thank you very much.
>> 
>> >Do as you like it. I do not think this is a bad approach.
>> 
>> I'll to that, then. I'm currently working on the D API, but I'll try
>> to get the copied deimos compatible project started in the next
>> couple of days.
>
> Sounds good. I'm going to write the documentation this evening.

Awesome, thanks.

Moritz Maxeiner wrote:
> On Monday, 18 March 2013 at 14:51:00 UTC, Jens Mueller wrote:
> >Moritz Maxeiner wrote:
> >>On Monday, 18 March 2013 at 13:50:29 UTC, Jens Mueller wrote:
> >>
> >>I'll to that, then. I'm currently working on the D API, but I'll
> >>try
> >>to get the copied deimos compatible project started in the next
> >>couple of days.
> >
> >Sounds good. I'm going to write the documentation this evening.
> 
> Awesome, thanks.

Took longer than expected.
http://jkm.github.com/phobos/deimos.html
I will post this on digitalmars.D whether it is possible to add it to
the official site and to collect feedback.

Jens