Moritz Maxeiner wrote:
> On Saturday, 23 March 2013 at 09:30:07 UTC, Jens Mueller wrote:
> >Moritz Maxeiner wrote:
> >>A couple of more things I forgot to mention:
> >>
> >>- You will need to additionally add bindings for all the LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}
> >>functions. They reside inside the target libraries and not
> >>inside
> >>llvm-c, which is why only translating the C API won't get you
> >>them.
> >>The LLVMInitializeNativeTarget function , which is necessary to
> >>use
> >>LLVM for jitting - otherwise you'll only have access to
> >>interpreting, which is horribly slow - uses them, but it is a
> >>static
> >>inline function and as such does not get exposed because LLVM
> >>builds
> >>with the option to hide all inline functions. That is not a
> >>problem
> >>for C, as this function gets defined in the header, but for D it
> >>is
> >>a big problem, because you cannot access
> >>LLVMInitializeNativeTarget.
> >>You need to recreate it in D by using all the functions it
> >>references. Of course, you'll also have to accomodate the fact
> >>that
> >>these referenced functions may or may not be compiled in
> >>depending
> >>on which target where selected when compiling LLVM.
> >
> >Do you happen to know why these functions are not declared in the
> >header
> >files. I mean when using the C interface in C I wouldn't even know
> >that
> >these functions exist.
> 
> I suspect it is because of LLVM's internal structure. Each supported
> target is presented by its own directory and they are afaict
> mutually independent from one another, "so their initialiation
> routines should be part of them themselves" I would guess, but I
> don't know, sorry.
> Also, when using the C interface in C there is little need to know
> them because they get encapsulated in LLVMInitializeAllTargets and
> LLVMInitializeNativeTarget.

Not sure whether this fixes the problem completely. Can you check https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/target.d#L157 I came up with this solution to replace the macros in the C binding.

Jens

On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller wrote:
> Moritz Maxeiner wrote:
>> On Saturday, 23 March 2013 at 09:30:07 UTC, Jens Mueller wrote:
>> >Moritz Maxeiner wrote:
>> >>A couple of more things I forgot to mention:
>> >>
>> >>- You will need to additionally add bindings for all the LLVMInitialize"TARGET_NAME"{TargetInfo,Target,TargetMC,AsmParser,AsmPrinter,Disassembler}
>> >>functions. They reside inside the target libraries and not
>> >>inside
>> >>llvm-c, which is why only translating the C API won't get you
>> >>them.
>> >>The LLVMInitializeNativeTarget function , which is necessary to
>> >>use
>> >>LLVM for jitting - otherwise you'll only have access to
>> >>interpreting, which is horribly slow - uses them, but it is a
>> >>static
>> >>inline function and as such does not get exposed because LLVM
>> >>builds
>> >>with the option to hide all inline functions. That is not a
>> >>problem
>> >>for C, as this function gets defined in the header, but for D it
>> >>is
>> >>a big problem, because you cannot access
>> >>LLVMInitializeNativeTarget.
>> >>You need to recreate it in D by using all the functions it
>> >>references. Of course, you'll also have to accomodate the fact
>> >>that
>> >>these referenced functions may or may not be compiled in
>> >>depending
>> >>on which target where selected when compiling LLVM.
>> >
>> >Do you happen to know why these functions are not declared in the
>> >header
>> >files. I mean when using the C interface in C I wouldn't even know
>> >that
>> >these functions exist.
>> 
>> I suspect it is because of LLVM's internal structure. Each supported
>> target is presented by its own directory and they are afaict
>> mutually independent from one another, "so their initialiation
>> routines should be part of them themselves" I would guess, but I
>> don't know, sorry.
>> Also, when using the C interface in C there is little need to know
>> them because they get encapsulated in LLVMInitializeAllTargets and
>> LLVMInitializeNativeTarget.
>
> Not sure whether this fixes the problem completely. Can you check
> https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/target.d#L157
> I came up with this solution to replace the macros in the C binding.

It looks mostly okay to me with one problem: Afaict the code enforces the presence of the initialization routines of all targets, e.g. if one target is missing LLVMInitializeAllTargets will not link, as there are undefined references for that missing target, but LLVM may or may not be compiled with that target so you cannot enforce its presence in the bindings. For runtime loading the solution I used was to check the function pointer for null; for linking you have this problem: When using linking, knowing which targets are available happens at link time (when the LLVM libraries are linked in), which means you cannot use any compile time tricks for automatic detection of which targets are available.
The only solution for that problem I can think of would be to use runtime reflection and check at runtime for each initialiation routine if it is a callable function, but afaik D only has compile time reflection.

- Moritz

PS: All your recent posts seem to create new threads, I don't know why, though, just mentioning it because some people not interested in this might get annoyed.

On Saturday, 23 March 2013 at 08:53:47 UTC, Jens Mueller wrote:
> Moritz Maxeiner wrote:
>> I could add support for it via a version flag. E.g. set
>> -version=DEIMOS_LLVM or something similar. That way someone using
>> llvm-d can choose to either use the included C bindings or use your
>> deimos compatible ones. My only concern would be that the LLVM C
>> function signatures would have to be translated in the same, or at
>> least a compatible way in both my C bindings and the deimos ones, so
>> using either of them would just be a matter of which import
>> statement to use.
>> Since I've translated them pretty much strictly according to the
>> "interfacing with c" guideline that's probably the case already, but
>> I'd have to try it out first.
>
> I would like that. Probably the signatures are already the same. There
> is not much freedom when translating them.

I've updated llvm-d to do just that. When compiling with -version=DEIMOS_LLVM the D API will import the deimos-llvm bindings instead of llvm-d's own ones. I've not yet tested it fully, but it should work. One thing, though: Your
https://github.com/jkm/deimos-llvm/blob/master/deimos/llvm/c/executionengine.d
should also have executionengine as the module name, not enhanceddisassembly.

-Moritz

On 03/23/2013 05:01 AM, Moritz Maxeiner wrote:
[snip]
>
> No problem at all. There is an example quoted in the README and how to
> compile it, so without further information form your side I don't know
> what the problem is. To get the example from the README working:
> - Download/Clone the github repo into a folder (let's say llvm-d)
> - cd into that folder
> - execute rdmd -L-ldl sample/multithreaded.d
>
> The need for the -dl flag is only for POSIX platforms and will be gone
> with the next commit as I have included a lib pragma.
>
> If the above doesn't help could you please tell me what exactly you have
> done and at which point you have the problems?
>
> -- Moritz

...And its fixed.
TY

Also, you named the enums, meaing "LLVMCCallConv", like in C, doesn't work and one has to use "LLVMCallConv.LLVMCCallConv". Same for the other enums.

On Saturday, 23 March 2013 at 14:42:06 UTC, 1100110 wrote:
> On 03/23/2013 05:01 AM, Moritz Maxeiner wrote:
> [snip]
>>
>> No problem at all. There is an example quoted in the README and how to
>> compile it, so without further information form your side I don't know
>> what the problem is. To get the example from the README working:
>> - Download/Clone the github repo into a folder (let's say llvm-d)
>> - cd into that folder
>> - execute rdmd -L-ldl sample/multithreaded.d
>>
>> The need for the -dl flag is only for POSIX platforms and will be gone
>> with the next commit as I have included a lib pragma.
>>
>> If the above doesn't help could you please tell me what exactly you have
>> done and at which point you have the problems?
>>
>> -- Moritz
>
> ...And its fixed.
> TY

No problem, I discovered that using "struct XYZ;" instead of "struct XYZ {};" will lead to forward referencing errors (which I did not expect) and changed all the structs accordingly. I'll have another sample up later today to show a more complex use-case.

On Saturday, 23 March 2013 at 10:01:19 UTC, Moritz Maxeiner wrote:
> No problem at all. There is an example quoted in the README and how to compile it, so without further information form your side I don't know what the problem is. To get the example from the README working:
> - Download/Clone the github repo into a folder (let's say llvm-d)
> - cd into that folder
> - execute rdmd -L-ldl sample/multithreaded.d
>
> The need for the -dl flag is only for POSIX platforms and will be gone with the next commit as I have included a lib pragma.
>
> If the above doesn't help could you please tell me what exactly you have done and at which point you have the problems?
>
> -- Moritz

Thanks so much.

I had dub upgrade the llvm-d package and now I have different problems. That said, I do have a working example (as long as rdmd is used).

I have updated the gist (with repro steps): https://gist.github.com/Zshazz/c7dbd6eee0b6b242252b

I'm running LLVM 3.2, so the example given in the README doesn't work. But the code in the gist works as long as you run 'dub --rdmd', so that's some progress! Not really sure about the linking problems without rdmd. This is sufficient to start some work with it to really try some things out, though.

I'll keep messing around with it. It seems like you already know that JIT doesn't work because we need InitializeNativeTarget.

That all said, the tutorials, instructions, and documentation for the C API of LLVM is pretty sparse. The best I've seen to figure out how things work is to go through http://llvm.org/doxygen/modules.html and piece together (very slowly) how something might work. I'm also having to cross reference tutorials made for the C++ API and figure out how to make it work in C by reading the raw source code to get the equivalent calls. Fortunately I did find that one post that gave a close-to-complete example in C. Is there any better way to do this at this point?

Thanks again.

On Saturday, 23 March 2013 at 19:57:52 UTC, Chris Cain wrote:
> On Saturday, 23 March 2013 at 10:01:19 UTC, Moritz Maxeiner wrote:
>> No problem at all. There is an example quoted in the README and how to compile it, so without further information form your side I don't know what the problem is. To get the example from the README working:
>> - Download/Clone the github repo into a folder (let's say llvm-d)
>> - cd into that folder
>> - execute rdmd -L-ldl sample/multithreaded.d
>>
>> The need for the -dl flag is only for POSIX platforms and will be gone with the next commit as I have included a lib pragma.
>>
>> If the above doesn't help could you please tell me what exactly you have done and at which point you have the problems?
>>
>> -- Moritz
>
> Thanks so much.
>
> I had dub upgrade the llvm-d package and now I have different problems. That said, I do have a working example (as long as rdmd is used).
>
> I have updated the gist (with repro steps): https://gist.github.com/Zshazz/c7dbd6eee0b6b242252b

Regarding the two warnings:
The second one for line 346 has been fixed.
I have no idea how you could get the first one as the statement is - at least according to my logic - reachable: It should be reached if the "isSizedDerivedType" is used on a StructType instance. I'll look into that shortly.

Other than that, the link time errors appear because the dl library (which allows for loading of dynamic libraries at runtime) did not get linked in. That should not be possible as llvm.util.shlib contains a lib pragma that ensures the dl library does get linked in.

Also, you're giving d string to c function which expect c strings. Not a good idea (c strings are \0 terminated, d strings aren't). Use std.string.toStringz (which allocates GC memory, though and the c strings may dissappear while LLVM is using them as D collects them, because the GC doesn't know about any use on the C side) or use llvm.util.memory.toCString (which does the same thing as toStringz, only it doesn't allocate GC memory, but "unmanaged" memory).

And you don't need the two lines with the lib directory if you have LLVM installed. That exists in the README sample only to show how it is done for people who wish to ship the LLLVM library together with their program inside a subfolder, I've removed them from the README example since it seems to cause confusion and also added the fibonacci example to the README to showcase something more complex.

>
> I'm running LLVM 3.2, so the example given in the README doesn't work. But the code in the gist works as long as you run 'dub --rdmd', so that's some progress! Not really sure about the linking problems without rdmd. This is sufficient to start some work with it to really try some things out, though.

There another example now available under samples/fibonacci.d that shows a complex example that works with LLVM 3.2.

>
> I'll keep messing around with it. It seems like you already know that JIT doesn't work because we need InitializeNativeTarget.

It does work, that is precisely what I reimplemented LLVMInitializeNativeTarget and LLVMInitializeAllTargets in D for. See llvm.c.functions.

>
> That all said, the tutorials, instructions, and documentation for the C API of LLVM is pretty sparse. The best I've seen to figure out how things work is to go through http://llvm.org/doxygen/modules.html and piece together (very slowly) how something might work. I'm also having to cross reference tutorials made for the C++ API and figure out how to make it work in C by reading the raw source code to get the equivalent calls. Fortunately I did find that one post that gave a close-to-complete example in C. Is there any better way to do this at this point?

You just pretty much described what I've been doing myself, I don't know of a better way at present, sorry. I'm trying to advance the D API to a point where it's usable and then I'll add decent documentation to the D API, but until then you'll have to suffer through the doxygen "documentation" the same way I do^^

- Moritz

The second warning has been dealt with now, as well (even though the program semantic has not changed at all, somethings wrong with the unreachable statement detenction >.>).

Also, readded the -ldl flag to lflags-posix in the package.json which solves the problem with the dl library not being linked in by dub. Apparently dub does not honor lib pragmas, you might want to report that (as you found it).

TLDR: Your example should now work, provided you fix what I previously mentioned. You can also look at sample/fibonacci.d which I used instead of your fac to confirm that you gist now works.

- Moritz

Moritz Maxeiner wrote:
> On Saturday, 23 March 2013 at 16:37:35 UTC, Jens Mueller wrote:
> >Moritz Maxeiner wrote:
> >>On Saturday, 23 March 2013 at 10:31:30 UTC, Jens Mueller wrote:
> >>
> >>It looks mostly okay to me with one problem: Afaict the code
> >>enforces the presence of the initialization routines of all
> >>targets,
> >>e.g. if one target is missing LLVMInitializeAllTargets will not
> >>link, as there are undefined references for that missing target,
> >>but
> >>LLVM may or may not be compiled with that target so you cannot
> >>enforce its presence in the bindings. For runtime loading the
> >>solution I used was to check the function pointer for null; for
> >>linking you have this problem: When using linking, knowing which
> >>targets are available happens at link time (when the LLVM
> >>libraries
> >>are linked in), which means you cannot use any compile time
> >>tricks
> >>for automatic detection of which targets are available.
> >>The only solution for that problem I can think of would be to
> >>use
> >>runtime reflection and check at runtime for each initialiation
> >>routine if it is a callable function, but afaik D only has
> >>compile
> >>time reflection.
> >
> >I wonder how they do it in C. Don't you have to set a macro?
> 
> Afaict they rely on the fact that when you install llvm on your system you get the {/usr/include/}llvm/Config/Targets.def file, in which is set what targets LLVM was compiled with (it gets generated at LLVM compile-time). Then the Target.h, in which the LLVMInitializeAllTargets function rests, includes that file and does some of that macro-voodoo that makes C/C++ so "lovable" to only create calls for the targets enabled in the Targets.def file when LLVMInitalizeAllTargets gets inlined. Of course, that solution isn't viable because afaik you cannot access filesystem IO functions in D CTFE, meaning the Targets.def file is useless to you. <rant>Hooray for the C/C++ preprocessor, may it die, die, die!</rant>

If I knew the path to the Targets.def file at compile I could load its
contents and use it.
http://dlang.org/expression.html#ImportExpression
Still have to find out how to get the path.

Jens