Darn. Good catch. Have an upvote :)

I was all primed to give an explanation about some of the internals of the language, and after checking ... that's not whats going on here. What I thought was happening (and why I wasn't overly surprised by the saxpy results) was this:

- The above data is actually from my M4 Max MBP, not the PC using a 3090/5080/both

- xc has the concept of memory that is sufficiently aligned that it can be used directly with GPU data in a unified memory model. You just annotate the pointer with 'simd:', thus:

   simd:u8* ptr = new u8[4096]

And what I had assumed was happening was that the C front-end was annotating its buffer correctly, that the kernel didn't need to do any buffer-copying (which is what generally slows down saxpy) and that the GPU was just ripping through the data in-place and handing back control to the CPU.

That ... turns out not to be the case [grin]. The CFE is not marking buffers that get allocated on the heap with the simd: annotation (this is a bug, I will fix it), so you are absolutely correct and the numbers as presented above don't stand up for saxpy. Here's a more in-depth table for saxpy, and it shows where the error crept in:

     ┌───────────────────────────────────┬───────────┐
     │   saxpy, M4 Max (C, --par-auto)   │   Time    │
     ├───────────────────────────────────┼───────────┤
     │ clang -O2, one thread             │ 1,398 µs  │
     ├───────────────────────────────────┼───────────┤
     │ xcc, all CPU threads              │ 518 µs    │
     ├───────────────────────────────────┼───────────┤
     │ xcc, auto (picks all CPU threads) │ 593 µs    │
     ├───────────────────────────────────┼───────────┤
     │ xcc, GPU                          │ 11,240 µs │
     └───────────────────────────────────┴───────────┘

In fact the GPU-run's own report shows where the GPU time goes. Each saxpy pass takes about 0.8ms on the GPU itself, but just over 11ms end to end. Almost all of the difference is copying the two 64 MB arrays to and from the GPU-aligned buffer on every run. The 2.1× over clang is just memory bandwidth: one core can't saturate the M4 Max's memory bus and 16 can.

The 'auto' mode (the default unless you pin a kernel as must-use-this-execution-style) measures a few iterations of the work-tiles at runtime, and picks the fastest to run the full job with on the local hardware at runtime (unless it has metrics from a previous run).

Bottom line: what is actually happening here is that the 'auto' mode (also in the original table header) is doing its measurement, and picking the fastest route - which is the "all CPU threads" mode. You can see the measurement cost (I always run these tests after clearing out the persisted previous run, so there's always the measurement-cost in the 'auto' figure). What was misleading was the "(GPU)" in the original table header, I have to apologise for that.

Having said that, I'm glad you asked - in this case I don't think it'd make an actual difference (multi-threaded CPU still beats the non-buffer-transfer GPU) but potentially an integrated GPU (Mac or one of the "AI" Ryzens perhaps) might benefit from another, very slightly more complex compute, now that I know the automatic memory-alignment isn't working properly :)

Glad my nitpick turned out to be useful!

Yeah, for an M4 Max I'd expect 1.5-2x (the CPU can only use about half the total bandwidth, roughly, so the GPU should still win if you discount initialization), though if memory serves the Metal API needed some special flag set in addition to the basic alignment requirement in order to actually directly use the same buffer without extra staging, at least when I last used it.

More generally, if your language builds a unified memory abstraction on top of systems that have physically heterogenous memory, make sure the cost model doesn't just include the nominal bandwidth differences (or asymptotic ones, if you do live profiling), but also the overhead of any extra staging buffers you use (explicitly or not, drivers and runtimes are not generally shy of allocating behind your back).

Yep, Metal has a 'noCopy' initialisation, otherwise it'll do a copy to aligned data first.

The costs are basically latency-time-based (time to prepare the system for execution) coupled with execution-time-based, rignt now. The thing is they're done by measurement, not prediction, so even if I manage to screw up the setup and do something non-optimally, that measurement is still a valid discriminant for the system it's actually running on - and recall that everything is done at runtime, not compile-time, so it's that system that's being characterised.

I do actually update the network latency timings on an ongoing basis - there's a heartbeat that sends back info on a periodic basis, and the error-condition mechanism has to cope with a node falling off the network etc.

And again, thanks for the insight :)