On Thursday, 20 January 2022 at 04:01:09 UTC, Araq wrote:

Given this endorsement I started reading up on Julia/GPU... Here are a few things that I found:
A gentle tutorial: https://nextjournal.com/sdanisch/julia-gpu-programming
Another, more concise: https://juliagpu.gitlab.io/CUDA.jl/tutorials/introduction/

For those that are video oriented, here's a recent workshop:
https://www.youtube.com/watch?v=Hz9IMJuW5hU

While I admit to just skimming that, very long, video I was impressed by the tooling on display and the friendly presentation.

In short, I found a lot to like about Julia from the above and other writings but the material on Julia AST macros specifically was ... underwhelming. AST macros look like an inferior tool in this low level setting. They are slightly less readable to me then the dcompute alternatives without offering any compensating gain in performance.

On Thursday, 20 January 2022 at 08:36:32 UTC, Ola Fosheim Grøstad wrote:

Because compilers are not sufficiently advanced to extract all the performance that is available on their own.

A good example of where the automated/simple approach was not good enough is CUB (CUDA unbound), a high performance CUDA library found here https://github.com/NVIDIA/cub/tree/main/cub

I'd recommend taking a look at the specializations that occur in CUB in the name of performance.

D compile time features can help reduce this kind of mess, both in extreme performance libraries and extreme performance code.

I think that that is an entirely reasonable goal but such transparency may cost performance and any such cost will be unacceptable to some.

Yes. Always good to have alternatives. Fully automated is one option, hinted is a second alternative, meta-programming assisted manual is a third.

If you can achieve your performance objectives with automated or hinted solutions, great! But what if you can't? Most people will not have to go as hardcore as the CUB authors did to get the performance they need but I find myself wanting more than the compiler can easily give me quite a bit. I'm very happy to have the meta programming tools to factor/reduce these "manual" programming task.

On Thursday, 20 January 2022 at 12:18:27 UTC, Bruce Carneal wrote:

Well, but D developers cannot test on all available CPU/GPU combinations either so then you don't know if SIMD would perform better than GPU.

Something automated has to be present, at least on install, otherwise you risk performance degradation compared to a pure SIMD implementation. And then it is better (and cheaper) to just avoid GPU altogether.

I am sure you are right, but I didn't find anything special when I browsed through the repo?

Well, my gut instinct is that if you want maximal performance for a specific GPU then you would be better off using Metal/Vulkan/etc directly?

But I have no experience with that as it is quite time consuming to go that route. Right now basic SIMD is time consuming enough… (but OK)

On Thursday, 20 January 2022 at 13:29:26 UTC, Ola Fosheim Grøstad wrote:

It can be very expensive to write and test all the permutations, yes, but often you'll understand the bottlenecks of your algorithms sufficiently to be able to correctly filter out the work up front. Restating here, these are a few of the traditional ways to look at it: Throughput or latency limited? Operand/memory or arithmetic limited? Power (watts) preferred or other performance?

It's possible, for instance, that you can know, from first principles, that you'll never meet objective X if forced to use platform Y. In general, though, you'll just have a sense of the order in which things should be evaluated.

Yes, SIMD can be the better performance choice sometimes. I think that many people will choose to do a SIMD implementation as a performance, correctness testing and portability baseline regardless of the accelerator possibilities.

The key thing to note is how much effort the authors put into specialization wrt the HW x SW cross product. There are entire subdirectories devoted to specialization.

At least some of this complexity, this programming burden, can be factored out with better language support.

That's what seems reasonable, yes, but fortunately I don't think it's correct. By analogy, you can get maximum performance from assembly level programming, if you have all the compiler back-end knowledge in your head, but if your language allows you to communicate all relevant information (mainly dependencies and operand localities but also "intrinsics") then the compiler can do at least as well as the assembly level programmer. Add language support for inline and factored specialization and the lower level alternatives become even less attractive.

Indeed. I'm currently working on the SIMD variant of something I partially prototyped earlier on a 2080 and it has been slow going compared to either that GPU implementation or the scalar/serial variant.

There are some very nice assists from D for SIMD programming: the __vector typing, __vector arithmetic, unaligned vector loads/stores via static array operations, static foreach to enable portable expression of single-instruction SIMD functions like min, max, select, various shuffles, masks, ... but, yes, SIMD programming is definitely a slog compared to either scalar or SIMT GPU programming.

On Thursday, 20 January 2022 at 17:43:22 UTC, Bruce Carneal wrote:

This doesn't change the desire to do performance testing at install or bootup IMO. Even a "narrow" platform like Mac is quite broad at this point. PCs are even broader.

My understanding is that the presentation Bryce made suggested that you would just write "fairly normal" C++ code and let the compiler generate CPU or GPU instructions transparently, so you should not have to write SIMD code. SIMD would be the fallback option.

I think that the point of having parallel support built into the language is not to get the absolute maximum performance, but to make writing more performant code more accessible and cheaper.

If you end up having to handwrite SIMD to get decent performance then that pretty much makes parallel support a fringe feature. E.g. it won't be of much use outside HPC with expensive equipment.

So in my mind this feature does require hardware vendors to focus on CPU/GPU integration, and it also requires a rather "intelligent" compiler and runtime setup in order to pay for the debts of the "abstraction overhead".

I don't think just translating a language AST to an existing shared backend will be sufficient. If that was sufficient Nvidia wouldn't need to invest in nvc++?

But, it remains to be seen who will pull this off, besides Nvidia.

On Thursday, 20 January 2022 at 19:57:54 UTC, Ola Fosheim Grøstad wrote:

Never meant to say that it did. Just pointed out that you can factor some of the work.

The dream, for decades, has been that "the compiler" will just "do the right thing" when provided dead simple code, that it will achieve near-or-better-than-human-tuned levels of performance in all scenarios that matter. It is a dream worth pursuing.

If accessibility requires less performance then you, as a language designer, have a choice. I think it's a false choice but if forced to choose my choice would bias toward performance, "system language" and all that. Others, if forced to choose, would pick accessibility.

I disagree but can't see how pursuing it further would be useful. We can just leave it to the market.

I put more faith in efforts that cleanly reveal low level capabilities to the community, that are composable, than I do in future hardware vendor efforts.

Well, at least for current dcompute users, it already is sufficient. The Julia efforts in this area also appear to be successful. Sean Baxter's "circle" offshoot of C++ is another. I imagine there are or will be other instances where relatively small manpower inputs successfully co-opt backends to provide nice access and great performance for their respective language communities.

I don't think there is much that remains to be seen here. The rate and scope of adoption are still interesting questions but the "can we provide something very useful to our language community?" question has been answered in the affirmative.

People choose dcompute, circle, Julia-GPU over or in addition to CUDA/OpenCL today. Others await more progress from the C++/SycL movement. Meaningful choice is good.

On Thursday, 20 January 2022 at 08:36:32 UTC, Ola Fosheim Grøstad wrote:

There are two major advantages for compile time features, for the host and for the device (e.g. GPU).

On the host side, D meta programming allows DCompute to do what CUDA does with its <<<>>> kernel launch syntax, in terms of type safety and convenience, with regular D code. This is the feature that makes CUDA nice to use and OpenCL's lack of such a feature quite horrible to use, and change of kernel signature a refactoring unto itself.

On the device side, I'm sure Bruce can give you some concrete examples.

You want it to be transparent, not invisible.

Wont work, D needs to be able to call the objective-c.
I mean you could use a C or C++ shim, but that would be pretty ugly.

On Friday, 21 January 2022 at 03:23:59 UTC, Nicholas Wilson wrote:

Are these resolved at compile time (before the executable is installed on the computer) or are they resolved at runtime?

I guess there might be instances where you might want to consider to change the entire data layout to fit the hardware, but then you to some extent outside of what most D programmers would be willing to do.

The goal is to make it look like a regular C++ library, no extra syntax.

Just write the whole runtime in Objective-C++. Why would it be ugly?

On Friday, 21 January 2022 at 08:56:22 UTC, Ola Fosheim Grøstad wrote:

Before. But with SPIR-V there is an additional compilation/optimisation step where it is converted into whatever format the hardware uses, also you could set specialisation constants here if I ever get around to supporting those. I think it probably also happens with PTX (which is an assembly like format) to whatever the binary format is.

Indeed.

There is an important difference between it looking like regular C++ (i.e. function calls not <<<>>>) and the compiler doing auto-GPU-isation. I'm not sure which one you're referring to here. I'm all for the former, that's what Dcompute does, the latter falls too far into the sufficiently advanced compiler and would have to necessarily determine what to send to the GPU and when, which could seriously impact performance.

Just. I mean it would be doable, but I rather not spend my time doing that.

On Friday, 21 January 2022 at 09:45:32 UTC, Nicholas Wilson wrote:

:-D This is where you need more than one person for the project…

I might do it, if I found a use case for it. I am sure some other contributor than yourself could do it if Metal support was in.