Does PGO work on Windows with LDC?

If so, how do I use it? Been trying to follow the information available but get various errors related to linker or runtime errors.

Is it supported on Windows?

Thanks

On Monday, 16 October 2023 at 17:00:41 UTC, Imperatorn wrote:

With msvc I get
unresolved external symbol llvm_profile_instrument_target referenced in function _D4hash7keynode9xtoHashFNbNeKxSQBfQBdZm
unresolved external symbol llvm_profile_runtime referenced in function llvm_profile_runtime_user

And with link-internally I get
lld-link: error: undefined symbol: llvm_profile_runtime
lld-link: error: undefined symbol: llvm_profile_instrument_target

On Monday, 16 October 2023 at 17:13:04 UTC, Imperatorn wrote:

I get the same on Linux:
undefined reference to `__llvm_profile_instrument_target'

And trying a minimal example gives me:

Exception Code: 0xC000001D
0x00007FF603511EDE, C:\Program Files\LDC 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x3201EDE byte(s)
0x00007FF60318F052, C:\Program Files\LDC 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2E7F052 byte(s)
0x00007FF6032A01DA, C:\Program Files\LDC 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F901DA byte(s)
0x00007FF60329FE2F, C:\Program Files\LDC 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F8FE2F byte(s)
0x00007FF6032A0135, C:\Program Files\LDC 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F90135 byte(s)
0x00007FF6031884DD, C:\Program Files\LDC 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2E784DD byte(s)
0x00007FF60353F4C4, C:\Program Files\LDC 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x322F4C4 byte(s)
0x00007FFF17AB7344, C:\windows\System32\KERNEL32.DLL(0x00007FFF17AA0000) + 0x17344 byte(s), BaseThreadInitThunk() + 0x14 byte(s)
0x00007FFF18F426B1, C:\windows\SYSTEM32\ntdll.dll(0x00007FFF18EF0000) + 0x526B1 byte(s), RtlUserThreadStart() + 0x21 byte(s)

On Monday, 16 October 2023 at 17:43:21 UTC, Imperatorn wrote:

After some experimentation it turned out it was working after all, but not with any code and dependencies.

For example in my case, if I removed asdf deserialize, it worked, but if I introduced it again, I got a bunch of errors like this

lld-link: error: undefined symbol: ClassInfo for asdf.asdf.AsdfSerdeException
>>> [...]

lld-link: error: undefined symbol: vtable for asdf.asdf.AsdfSerdeException
>>> [...]

lld-link: error: undefined symbol: initializer for asdf.asdf.AsdfSerdeException

On Monday, 16 October 2023 at 18:10:21 UTC, Imperatorn wrote:

For documentation purposes I will share the solution here.

The problem was I was using dub and didn't know how to forward everything correctly as dflags.

In your dflags, supply something like this, do not add -of=test or whatever, dub will overwrite it:

"dflags": ["-m64", "-release" ,"-fprofile-instr-generate"]

Then just:

dub build --compiler=ldc2

This will produce an executable with the same name as you specified in your dub config.

Now you run that and it will produce an output file that you will convert.

The rest of the steps can be read here:

https://johanengelen.github.io/ldc/2016/07/15/Profile-Guided-Optimization-with-LDC.html

On Tuesday, 17 October 2023 at 08:59:52 UTC, Imperatorn wrote:

If you are having trouble using the above, try the following instead:

dub build --compiler=ldc2 --build-mode=allAtOnce --combined

The complete process for dub (replace {programname} with your program name):

1. Specify a name and dflags

	"dflags": ["-fprofile-instr-generate"],
	"name": "{programname}",

2. Build

dub build --compiler=ldc2 --build-mode=allAtOnce --combined

3. Execute

./{programname}

4. Convert

ldc-profdata merge default.profraw -output converted.profdata

5. Change your dub.json to use the instrumented configuration and change the name

	"dflags": ["-fprofile-instr-use=converted.profdata"],
	"name": "{programname}_optimized",

6. Build

dub build --compiler=ldc2 --build-mode=allAtOnce --combined

7. Enjoy your PGO executable

All the above steps in a script, but without dub, assuming app.d is in source

Windows:

echo "Building app_normal.exe for comparison"
ldc2 -of="app_normal.exe" -O -g -gc -Isource source\app.d

echo "Building app_instrumented.exe with profiling"
ldc2 -fprofile-instr-generate -O -g -gc -of="app_instrumented.exe" -Isource source\app.d

echo "Running app_instrumented.exe"
app_instrumented.exe

ldc-profdata merge default.profraw -output converted.profdata

echo "Building PGO executable"
ldc2 -fprofile-instr-use=converted.profdata -O -of="app_optimized.exe" -Isource source\app.d -flto=full -defaultlib=phobos2-ldc-lto,druntime-ldc-lto

*nix:

echo "Building app_normal for comparison"
ldc2 -of="app_normal" -O -g -gc -Isource source/app.d

echo "Building app_instrumented with profiling"
ldc2 -fprofile-instr-generate -O -g -gc -of="app_instrumented" -Isource source/app.d

echo "Running app_instrumented"
./app_instrumented

ldc-profdata merge default.profraw -output converted.profdata

echo "Building PGO executable"
ldc2 -fprofile-instr-use=converted.profdata -O -of="app_optimized" -Isource source/app.d -flto=full -defaultlib=phobos2-ldc-lto,druntime-ldc-lto

On Tuesday, 17 October 2023 at 13:00:54 UTC, Imperatorn wrote:

Is there any way to incorporate some of what you figured out into
https://wiki.dlang.org/LDC_LLVM_profiling_instrumentation

It might also make sense to file an issue with dub about streamlining the process. I only see one issue currently that references PGO.
https://github.com/dlang/dub/issues/951

On Tuesday, 17 October 2023 at 13:00:54 UTC, Imperatorn wrote:

A typo:

It should be

-fprofile-instr-use="converted.profdata"

not

-fprofile-instr-use=converted.profdata

On Tuesday, 17 October 2023 at 13:42:24 UTC, jmh530 wrote:

Put another link here. For future "new documentation project":
https://archive.fosdem.org/2017/schedule/event/ldc_d_optimization/attachments/slides/1819/export/events/attachments/ldc_d_optimization/slides/1819/FOSDEM_2017.pdf

On Saturday, 21 October 2023 at 11:31:54 UTC, Sergey wrote:

Thanks for the links!