Thread overview
Need help with Windows linkage ( DMD using ImportC)
March 04
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\bin\Hostx64\x64\link.exe
DMD64 D Compiler v2.107.0

I'm doing some experimentation with ImportC on Windows, and getting this when linking.

blah.obj: error LNK2019: unresolved external symbol _mul128 referenced in function MultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol __shiftright128 referenced in function MultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol _umul128 referenced in function UnsignedMultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol __stosb referenced in function RtlSecureZeroMemory
blah.obj: error LNK2019: unresolved external symbol __readgsqword referenced in function NtCurrentTeb
blah.obj: error LNK2019: unresolved external symbol __imp_MapViewOfFileNuma2 referenced in function MapViewOfFile2
blah.obj: error LNK2019: unresolved external symbol __imp_CharUpperW referenced in function ua_CharUpperW

The details don't seem to matter much, windows headers #included in C files compiled in with ImportC bring these sorts of things in it seems, and I don't know how to link them, or exclude their declarations with a #define before I #include such.

March 05

On Monday, 4 March 2024 at 21:21:20 UTC, Carl Sturtivant wrote:

>
blah.obj: error LNK2019: unresolved external symbol _mul128 referenced in function MultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol __shiftright128 referenced in function MultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol _umul128 referenced in function UnsignedMultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol __stosb referenced in function RtlSecureZeroMemory
blah.obj: error LNK2019: unresolved external symbol __readgsqword referenced in function NtCurrentTeb
blah.obj: error LNK2019: unresolved external symbol __imp_MapViewOfFileNuma2 referenced in function MapViewOfFile2
blah.obj: error LNK2019: unresolved external symbol __imp_CharUpperW referenced in function ua_CharUpperW

I forced linkage of these unused symbols as follows, but it would be nice to have a clean way to proceed.

extern(C) {
    int _InterlockedExchangeAdd(int* Addend, int Value) { return 0; };
    long _InterlockedExchangeAdd64(long* Addend, long Value) { return 0; }
    void _mul128() {};
    void __shiftright128() {};
    void _umul128() {};
    void __stosb() {};
    void __readgsqword() {};
    void __imp_MapViewOfFileNuma2() {};
    void __imp_CharUpperW() {};
}

I got the D signatures of the first two so as to generate the correct linkage by using ImportC to translate the inclusion of Windows.h into a .di file, and searching.

March 07

On Tuesday, 5 March 2024 at 00:20:36 UTC, Carl Sturtivant wrote:

>

On Monday, 4 March 2024 at 21:21:20 UTC, Carl Sturtivant wrote:

>
blah.obj: error LNK2019: unresolved external symbol _mul128 referenced in function MultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol __shiftright128 referenced in function MultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol _umul128 referenced in function UnsignedMultiplyExtract128
blah.obj: error LNK2019: unresolved external symbol __stosb referenced in function RtlSecureZeroMemory
blah.obj: error LNK2019: unresolved external symbol __readgsqword referenced in function NtCurrentTeb
blah.obj: error LNK2019: unresolved external symbol __imp_MapViewOfFileNuma2 referenced in function MapViewOfFile2
blah.obj: error LNK2019: unresolved external symbol __imp_CharUpperW referenced in function ua_CharUpperW

I forced linkage of these unused symbols as follows, but it would be nice to have a clean way to proceed.

extern(C) {
    int _InterlockedExchangeAdd(int* Addend, int Value) { return 0; };
    long _InterlockedExchangeAdd64(long* Addend, long Value) { return 0; }
    void _mul128() {};
    void __shiftright128() {};
    void _umul128() {};
    void __stosb() {};
    void __readgsqword() {};
    void __imp_MapViewOfFileNuma2() {};
    void __imp_CharUpperW() {};
}

I got the D signatures of the first two so as to generate the correct linkage by using ImportC to translate the inclusion of Windows.h into a .di file, and searching.

This looks like a combination of two issues:

  1. Missing import libraries for Win32 API functions. Anything starting with __imp_ is a symbol that should be provided by a DLL import library. MapViewOfFileNuma2 for example is provided by onecore.lib in the Windows SDK, according to Microsoft documentation.

  2. C code referring to MSVC-specific compiler intrinsics. At least InterlockedExchangeAdd, InterlockedExchangeAdd64 and _stosb are such intrinsics. This is harder to resolve. There are two ways forward here: either implement a shim function that replicates the intrinsic's functionality if possible or add support for these intrinsics to DMD.

March 09

On Thursday, 7 March 2024 at 18:14:32 UTC, Gregor Mückl wrote:

>
  1. Missing import libraries for Win32 API functions. Anything starting with __imp_ is a symbol that should be provided by a DLL import library. MapViewOfFileNuma2 for example is provided by onecore.lib in the Windows SDK, according to Microsoft documentation.

Onecore: not sure what I did to add that dependency. Two symbols there. Thanks for the _imp clue.

>
  1. C code referring to MSVC-specific compiler intrinsics. At least InterlockedExchangeAdd, InterlockedExchangeAdd64 and _stosb are such intrinsics. This is harder to resolve. There are two ways forward here: either implement a shim function that replicates the intrinsic's functionality if possible or add support for these intrinsics to DMD.

Yes, not sure what the potential consequences are for my dirty replacement. Presumably DMD itself won't generate code using these, but ...

March 10

On Thursday, 7 March 2024 at 18:14:32 UTC, Gregor Mückl wrote:

>
  1. C code referring to MSVC-specific compiler intrinsics. At least InterlockedExchangeAdd, InterlockedExchangeAdd64 and _stosb are such intrinsics. This is harder to resolve. There are two ways forward here: either implement a shim function that replicates the intrinsic's functionality if possible or add support for these intrinsics to DMD.

Thanks for the explanation, on the strength of which I found a way to deal with this correctly.

I made intrinsics1.c that has #include <intrin.h> and contains an actual function for each missing intrinsic; e.g. for the missing __shiftright128 it has

unsigned __int64 D__shiftright128(
   unsigned __int64 LowPart,
   unsigned __int64 HighPart,
   unsigned char Shift
)
{ return __shiftright128(LowPart, HighPart, Shift); }

where I got the prototypes from this list of intrinsics.

Compiling this with MSVC cl -c intrinsics1.c produces a COFF object intrinsics1.obj containing an actual function to link to for each intrinsic. So this stage writes the code so we don't have to.

As a matter of necessity, the names of the functions in intrinsics1.c representing the MSVC intrinsics are not the same as their actual names. By my convention above they are prefixed with "D". Now we could simply write an extern(C) D function in a module say vcintrinsics.d, that has exactly the intrinsic's name, and calls the "D" prefixed function linked from intrinsics1.obj, e.g. for the above example vcintrinsics.d could contain

extern(C):

extern ulong D__shiftright128(ulong LowPart, ulong HighPart, ubyte Shift);

ulong __shiftright128(ulong LowPart, ulong HighPart, ubyte Shift){
    return D__shiftright128(LowPart, HighPart, Shift);
}

As DMD doesn't know of these intrinsics, it won't complain about defining a function with exactly the same name as an intrinsic so as to implement that intrinsic as an actual function, solving the problem. dmd -lib vcintrinsics.d intrinsics1.obj then produces a library that resolves the linkage issue.

An alternative not involving having two function calls to implement an intrinsic is to compile intrinsics1.c with MSVC into a DLL, and using a DEF file that renames the exports just like part of the solution here make those functions available under their original names when linking to its import library.

A lot of both of the above is boilerplate and can be automated. It would be even nicer if the MSVC tools could be persuaded to proceed in a similar way with DEF file renaming when building a static library, but I have not succeeded in making this happen. Anyone?

March 20

I found a way to make a solution for 64 bit Windows mechanically with many MSVC intrinsics, using only mingw64.
Here's an MSYS2 bash script.

gcc -E -P intrin.c -o vcintrinsics.c

sed -i 's/extern __inline__ __attribute__((__always_inline__,__gnu_inline__))//g' vcintrinsics.c

gcc -fPIC -shared -o vcintrinsics.dll vcintrinsics.c -Wl,--export-all-symbols -Wl,--output-def=vcintrinsics.def

#lib -nologo -machine:x64 -def:vcintrinsics.def -out:vcintrinsics.lib

dlltool -D vcintrinsics.dll -d vcintrinsics.def -l vcintrinsics.lib -m i386:x86-64

cp vcintrinsics.dll /c/D/dmd2/windows/bin64/
cp vcintrinsics.lib /c/D/dmd2/windows/lib64/

The commented out line is using the MS librarian to do the same job as the mingw64 libtool. This script builds a dll containing 100+ intrinsics defined in mingw64 and makes an import library for DMD to use it from a developer command prompt.

intrin.c contains only #include <intrin.h> and is preprocessed in the first line into vcintrinsics.c which if examined contains working definitions of many intrinsics but with extern __inline__ __attribute__((__always_inline__,__gnu_inline__)) prefixing them, so they will not compile to actual library functions. The sed command strips those prefixes out, and the compilation makes a DLL containing the the intrinsics and a DEF file listing their names for linkage, ready for dlltool to assemble an import library suitable for DMD to link to so as to be able to dynamically link the DLL.

And that's it; so far the result has just worked and I've had no linkage issues due to missing MSVC intrinsics since. I just put vcintrinsics.lib on the end of the dmd command line whenever I'm using real Windows headers in ImportC and the linker has always been satisfied so far. When I'm done, I can remove vcintrinsics from the dmd command line, and find out which intrinsics are actually needed for linking, and using the technique earlier in the thread above, satisfy those directly, so the result doesn't need vcintrinsics.dll.

Hopefully DMD will soon know about these MSVC intrinsics but until then this is an effective way to more-or-less permanently work around the inevitable linkage problems.