Add private WGP output validation and record measured K experiments

docs/windows-performance.md

@@ -337,8 +337,8 @@ over 1935 presents. ADLX's sixty polls report mean usage **93.183%**, core
 clock **3124.667 MHz**, board power **316.717 W**, temperature 57 C and hotspot
 82.9 C. Mean PresentMon GPU busy/wait is 14.551/0.800 ms. All presents retain
 Independent Flip / SyncInterval 0. This records high device-wide driver usage
-in this run; the previous user-reported 63% has not been captured from the same
+in this run; the user identified the earlier 63% reading as Task Manager.
-sensor/window and its discrepancy is unresolved. Do not infer a physical CPU
+It is not the same measurement as ADLX device-wide usage. Do not infer a physical CPU
 limit, isolated shader occupancy or a FSR4 speed comparison. Query overhead
 is median 0.094 ms per 500 ms poll, with a roughly 12 ms first query. Results:
 `test-results/silent-hill2-k-telemetry-unprofiled-4k`. The game stability
@@ -355,10 +355,13 @@ removes most conversion instructions (1066 static F16/F32 conversions become
 two) and reduces main-kernel VGPRs from 218 to 201; LDS remains 24576 bytes,
 with zero spills. These are compiler/ISA counts, not GPU time or occupancy.
 The candidate is rejected before moving to another layer: its real replay
-changes 1182727 half codes in the output allocation, max absolute difference
+changes 1187603 half codes in the output allocation, max absolute difference
 0.0390625 from the validated baseline. All values are finite. Against the
-current twelve-window NumPy reference, PSNR drops from baseline 78.7702 dB
+current twelve-window NumPy reference, PSNR drops from baseline 93.7527 dB
-to 69.2377 dB; max absolute error rises from 0.009765625 to 0.0390625.
+to 69.2377 dB; max absolute error rises from 0.0078125 to 0.0390625.
+The original comparison incorrectly selected a pre-normalization-fix saved
+output (78.7702 dB); a fresh replay matches `k-real-strict-dual-reference/enc1`
+exactly and supplies the corrected baseline here. The rejection still holds.
 Two independent PTX windows give 70.4022/70.08 dB. These pass the existing
 coarse reference thresholds but do not satisfy this optimization's baseline
 preservation gate. No game installation or performance claim follows.
@@ -390,6 +393,173 @@ and `gameOptimizationAccepted`. The second compiles the rejected experiment
 in a separate directory and generates no override manifest. Intrinsic API:
 [Clang gfx12 FP16 WMMA](https://clang.llvm.org/docs/AMDGPUBuiltinReference.html#builtin-amdgcn-wmma-f16-16x16x16-f16-w32-gfx12).
 
+### Controlled native replay timings
+
+`native_replay_probe` now supports `--benchmark-timing events|dispatch` and
+`--benchmark-batch 1..256`. Dispatch mode uses the documented
+`hipExtModuleLaunchKernel` start/stop events, with work-item dimensions derived
+from each module's launch metadata and flags zero. It does not change wave
+order or arithmetic. Control/candidate can have different native block/grid
+metadata; each launch uses its own geometry. The default remains event markers.
+
+Each AB/BA sample restores captured inputs outside timing, submits a series,
+waits for its end event, and logs both the event interval and independent host
+launch-to-completion interval divided by launch count. These are isolated
+diagnostics: batch intervals include dispatch/queue gaps and are not exclusive
+instruction execution times, frame latency, or game FPS. Zero, negative,
+nonfinite or event intervals exceeding the enclosing host interval are retained
+as `D4R_REPLAY_PROFILE_INVALID` and excluded. Reports keep timing sources,
+batch counts and log files separate, retaining host statistics and invalid
+samples; no partially valid pair is accepted by the benchmark runner.
+
+The new runner first executes fresh ordinary control/candidate replays and
+compares every allocation's SHA256 before timing. It stops at any mismatch,
+then checks the control output again after every benchmark. A successful
+timing run does not replace the independent NumPy/PTX correctness gates.
+Captured weights/IR and GPU outputs remain private.
+
+```powershell
+.\scripts\windows\benchmark-native-replay.ps1 -Module "$PWD\build\native-k-gfx1201\dltss_pwin_enc1_layer.hsaco" -Candidate "$PWD\build\native-k-gfx1201\dltss_pwin_enc1_layer.hsaco" -FixtureDirectory "$PWD\test-results\ngx-k-fp16-fenced-capture\replay\replay-000001-dltss_pwin_enc1_layer"
+```
+
+The same-module enc1 control has all exact allocations and 32 valid pairs per
+mode. Median per-launch times at the captured 512x288 workload are 0.054850 ms
+with separate single-launch events and 0.089700 ms with single-launch dispatch
+events. At batch 64 they converge to 0.048213/0.048017 ms. Thus dispatch events
+alone do not remove timing overhead; use the batch/host cross-check. All eleven
+K same-module controls pass sixteen pairs at batch 64 in both modes, including
+dec4. This does not fix or explain the previously recorded negative ZLUDA
+timestamps. Results: `test-results/k-replay-timing-methods` and
+`test-results/k-all-layers-batch-profile`.
+
+One-frame standalone K/4K HDR captures provide actual enc0/enc1/dec0 inputs.
+Each harness run uses the eleven validated native layers and private output
+tail, passes the finite-output scan and completes without network fallback. At this
+synthetic resolution enc1 uses a 61x35 grid; this workload is distinct from
+the game's 2259x1271 input. Experiments retain F32 WMMA instructions and strict
+FP16 rounding. `build-native-k-experiment.ps1` builds separate objects and no
+override manifest; no experimental object is installed by the game runner.
+
+CU/WGP and non-temporal load/store/both on enc1 preserve every allocation but
+do not improve its paired 4K timings. Position-only token tiling also preserves
+outputs: enc1 with two tiles and shared V spills four VGPRs and is about 9%
+slower; one-tile shared V has no measured gain. The two-tile shared-V enc0 run
+initially has a roughly 4% lower median paired ratio amid scheduling variance.
+A repeat with 64 AB/BA pairs at batch 64 gives control/candidate medians
+0.297060/0.282559 ms (events) and 0.299988/0.285011 ms (dispatch), with median
+paired ratios 0.950863/0.952240. A 512x288 replay independently gives ratios
+0.950396/0.952582. Both tested dec0 shared-V variants are slower or indistinguishable.
+
+Enc0 now selects shared V for the gfx12 strict baseline, preserving the existing
+two-token tiles, F32 WMMA and FP16 rounding. Gfx11 and arithmetic experiments
+retain their prior selection. `D4R_K_PRIVATE_V` restores the previous enc0 for
+comparison; the build helper exposes `-PrivateValues`. The exact CMake-produced
+object also passes a fresh 4K replay and 32 paired batch-64 samples, with ratios
+0.955418/0.948494. Its ELF hash differs from the experimental object; `.text`
+and `.rodata` match exactly, and numerical validation is repeated on the actual
+object. The other fifteen ordinary K/M objects remain byte-identical.
+Gfx1101 enc0/enc1 compile-only controls retain identical code and constant
+sections to their original sources; no RDNA3 hardware execution is claimed.
+
+Full K controls/candidates complete sixteen 4K frames, and twelve more with
+the game's 2259x1271 input and 3840x2160 output. Every compared RGB half code
+matches, all RGBA scans are finite. Results: `test-results/k-enc0-shared-full-4k`,
+`k-enc0-shared-full-game-input`, `k-4k-enc0-pos2-shared-repeat`,
+`k-enc0-pos2-shared-small` and `k-enc0-default-shared-4k`. The saving is about
+0.015 ms for enc0 in the isolated 4K replay, not 5% application FPS.
+
+The separate `D4R_K_PACKED_ACC` experiment stores already-rounded accumulators
+as half vectors between steps, widening for each unchanged F32 WMMA and
+rounding its F32 output back to half. The ordinary `acc8v` alias remains F32.
+Enc0/enc1/dec0 4K replays match every allocation exactly. Enc1 main VGPRs fall
+218 -> 195, LDS remains 24576 bytes and spills stay zero, but paired GPU time
+does not improve; enc0 is about 4% slower and dec0 about 1% slower. Combining
+packed storage with enc1 two-tile shared V is about 13% slower. The flag remains
+off. Smaller register counts alone are not evidence of an optimization.
+Raw results are under `test-results/k-4k-*-packed-acc` and
+`test-results/k-4k-enc1-pos2-shared-packed`.
+
+A direct recompilation of the private output tail's optimized LLVM IR with
+TheRock Clang 24 fails in instruction selection with `unsupported library
+call operation` (driver exit 1, backend exit 70, exception 0xE0000046). The generated crash
+reproducer/log are retained locally; strict FP is not removed to work around
+this. The working LLVM 22 output object remains selected. The HDR tail's
+metadata is wave32, 133 VGPRs, 6912 LDS bytes and no spills; LDR uses 128 VGPRs
+with the same LDS and no spills. These are static resource counts, not actual
+occupancy. Compiler scheduling hints are documented in the
+[LLVM AMDGPU backend guide](https://llvm.org/docs/AMDGPUUsage.html#llvm-ir-attributes).
+
+### Private output-tail WGP experiment
+
+ZLUDA source `fb0adc8714f57647320f0601533ec50331e378b0`, patch 0018, adds
+opt-in `D4R_ZLUDA_WGP=1`. It changes only CU/WGP placement for gfx11/gfx12,
+preserving wave size and arithmetic. The ordinary runtime stays in CU mode.
+The offline texture builder accepts `-ShaderMode wgp` and verifies `-cumode`
+on the emitted kernel before accepting the object. The compiled output-tail
+objects carry this mode independently of the runtime's environment variable.
+These NVIDIA-derived objects remain private; the public package excludes them.
+
+The matched K/4K comparison uses identical public network objects and the
+existing validated CU output kernels as control. Both HDR and LDR complete
+eight frames per variant; repeating with candidate-first order completes
+sixteen frames per variant. All 96 control/candidate frames are finite, and
+all 48 candidate RGB outputs exactly match their fresh controls. Every frame
+has a native output-tail hit and a valid serializing event measurement.
+
+| Output-tail median, ms | CU, first run | WGP, first run | CU, reverse order | WGP, reverse order |
+| --- | ---: | ---: | ---: | ---: |
+| LDR | 1.864650 | 1.703100 | 1.840260 | 1.687500 |
+| HDR | 2.012301 | 1.883395 | 2.016020 | 1.869250 |
+
+These are separate-process serializing diagnostic measurements, not paired
+resident-module timings or game FPS. Unchanged network kernels also vary
+between processes. The repeat indicates a candidate worth testing in the
+game; it does not establish a frame-rate improvement. Results:
+`test-results/k-output-wgp-reference-4k-clean-env` and
+`k-output-wgp-reference-4k-reverse`.
+The game-sized WGP async burst/recreate gate completes twelve frames, all
+finite; all six async RGB outputs match fresh sync controls exactly, with
+no CPU image copy or aged output (`k-output-wgp-async-game-input`).
+The exact CMake shared-V enc0 and WGP output tails together also pass twelve
+game-sized async/burst/recreation frames, with all six async RGB images exact
+(`k-enc0-shared-wgp-async-game-input`). Default selection of private output
+objects remains CU until a game comparison establishes the candidate's effect.
+
+```powershell
+.\scripts\windows\build-zluda-windows.ps1 -InstallRoot "$PWD\dist\zluda-windows-wgp-experiment"
+.\scripts\windows\build-native-texture.ps1 -DlssDll "$PWD\nvngx_dlss.dll" -ZludaRoot "$PWD\dist\zluda-windows-wgp-experiment" -ShaderMode wgp -OutputDirectory "$PWD\build\private-textures-wgp-gfx1201"
+.\scripts\windows\build-native-texture.ps1 -DlssDll "$PWD\nvngx_dlss.dll" -ZludaRoot "$PWD\dist\zluda-windows-wgp-experiment" -ShaderMode wgp -Kernel hiluma_engine_output_depthreg_mvhi_ldr_max_v2_rel -OutputDirectory "$PWD\build\private-textures-wgp-gfx1201"
+.\scripts\windows\test-native-texture.ps1 -TextureRoot "$PWD\build\private-textures-wgp-gfx1201" -ControlTextureRoot "$PWD\build\private-textures-gfx1201" -ZludaRoot "$PWD\dist\zluda-windows-deferred-profile" -OutputResolution 3840x2160
+```
+
+The compiler placement unit passes, all seventeen hardware/software CTest gates
+pass on the clean source build with ordinary CU defaults, and the six ZLUDA
+gates also pass with WGP explicitly enabled. All seventeen patches (0002..0018)
+apply to the clean pinned base `ee2f25a`. The numerical benchmark's rejected
+F16 candidate stops after its two ordinary reference replays with differing
+output and no timing samples (`k-replay-rejected-f16-gate-fixed`).
+
+### Diagnostic environment and input dimensions
+
+Some PowerShell/.NET hosts preserve removed process variables as empty strings.
+Presence-based Rust diagnostic flags then select deferred profiling, while
+empty codegen values can disable WMMA. The probe/game child environments now
+omit empty `D4R_` keys; the probe restores codegen defaults for null or empty
+values. Disabled WGP is also omitted so older runtimes keep their warm cache.
+The texture test requires valid timing records instead of silently accepting
+a profiling-free comparison. The initial candidate attempt times out during
+NGX initialization with no accepted frame; later pre-fix comparisons have
+correct RGB but no accepted GPU timings. Their logs remain in
+`k-output-wgp-reference-small`, `k-output-wgp-reference-small-fixed` and
+`k-output-wgp-reference-4k-fixed`; they are excluded from the table above.
+
+The D3D12 harness now supports `--ngx-input-resolution WIDTHxHEIGHT`, exposed
+as `-NgxInputResolution` in the probe runner and `-InputResolution` in the
+texture/async gates. Output defaults and limits remain unchanged. This allows
+the game's non-aligned input dimensions without inventing a larger output.
+An initial attempt using 4518x2542 output is rejected by the harness's existing
+range guard before GPU initialization; it is not a GPU or enc0 failure.
+
 Further game coverage (HIP kernel profiling disabled, GPU output checks enabled):
 M at 1920x1080 completes 4833 frames, all finite. K at 3840x2160 completes
 4685 frames, all finite. Neither has a backend failure, CPU image copy or an

docs/windows-rdna4-port.md

@@ -17,21 +17,52 @@ explicit limits; the validated package keeps conservative arithmetic.
 
 ## Milestone and gates
 
-**Current milestone: K performance remains open.** Async submission improves
+**Current milestone: K performance remains open.** The latest unprofiled 4K
-the user's 4K Silent Hill 2 reading to 62 FPS / 63% GPU utilization. A subsequent
+Silent Hill 2 scene capture records 64.661 FPS. Functional K/M and the
-controlled 30-second foreground-scene capture records 63.574 FPS, compared with roughly 80+
+reproducible archive do not close this gate. Prioritize measured K kernel cost
-FPS / 100% GPU utilization with FSR4 on this system. Functional K/M and the
+and retain queue/synchronization checks; postpone further M/FP8 optimization.
-reproducible archive do not close this gate. Prioritize K queue scheduling and
+The user's approximately 80+ FPS with FSR4 is not a captured same-scene control.
-host synchronization; postpone further M/FP8 optimization. These user readings
-are not a controlled same-scene benchmark. Do not infer a hardware CPU limit
-from utilization alone.
 
 A subsequent unprofiled K/4K capture records **64.661 FPS** with direct AMD
 ADLX mean usage **93.183%**, core clock **3124.667 MHz** and board power
-**316.717 W**. The previous 63% reading is from a different sensor/window;
+**316.717 W**. The user identified the previous 63% reading as Task Manager,
-its discrepancy remains unresolved. This new capture places GPU kernel cost
+which is distinct from ADLX's device-wide metric. This capture places GPU kernel cost
 at the center of the next K investigation, while retaining queue measurements.
 
+Native replay benchmarks now compare fresh exact allocations before timing,
+support event/dispatch profiling and paired batches, retain independent host
+completion intervals, and reject invalid or incomplete timings. Eleven K
+same-module controls pass both timing modes. Real K/4K enc0/enc1/dec0 captures
+and CU/WGP, cache, token-tiling and packed-storage experiments are recorded in
+[windows-performance.md](windows-performance.md#controlled-native-replay-timings).
+Packed storage lowers enc1 VGPRs but does not speed it up. Enc0's shared-V
+implementation preserves every tested output and gives about a 5% isolated
+layer saving at both small and 4K workloads; the strict gfx12 baseline now
+selects it. Other K layers, M and arithmetic remain unchanged. The previous direct-F16 experiment's
+saved-control selection was corrected to the current strict-normalization
+baseline; it remains rejected. K performance is still an open gate.
+
+The private output-tail WGP candidate gives a lower event median in both
+control-first and candidate-first K/4K tests, with exact RGB across 96 finite
+control/candidate frames. This is a separate-process diagnostic result, not
+game FPS. ZLUDA source `fb0adc8714f57647320f0601533ec50331e378b0` is rebuilt
+cleanly in `dist/zluda-windows-wgp-experiment`; optional patch 0018 preserves
+CU defaults and wave/arithmetic semantics. All seventeen CTest gates pass
+with defaults, and all six ZLUDA gates also pass with WGP explicitly enabled.
+All seventeen patches (0002..0018) apply to clean base `ee2f25a`.
+
+Enc0 full-frame controls pass at 1920x1080 and the game's 2259x1271 input,
+both outputting 3840x2160: 28 frames finite, all 14 candidate RGB images exact.
+The exact CMake enc0 object passes a fresh replay/paired benchmark; the other
+fifteen ordinary K/M objects retain their prior SHA256. The combined enc0 +
+WGP-output async/burst/recreation gate passes twelve further game-sized frames
+with all six async RGB images exact (`k-enc0-shared-wgp-async-game-input`).
+`-InputResolution` exposes actual render dimensions without changing the
+harness's output bounds. Empty task environment variables are removed from
+probe/game children so disabled diagnostic flags stay disabled; GPU comparisons
+now require actual valid timing samples. Commands, rejected experiments and
+the initial diagnostic/cache failures are recorded in `windows-performance.md`.
+
 The next diagnostics add `-ProfileCudaApi` (ZLUDA patch 0015): aggregate host
 CUDA API durations without HIP-event waits. Boundary stages now separate
 prepare, copy-list setup, input/output submission and output-copy drain.

patches/zluda/0018-opt-in-rdna-wgp-placement.patch

@@ -0,0 +1,93 @@
+From fb0adc8714f57647320f0601533ec50331e378b0 Mon Sep 17 00:00:00 2001
+From: xdfnx-dev <[email protected]>
+Date: Fri, 2 Oct 2026 14:40:56 +0300
+Subject: [PATCH] Add opt-in RDNA CU/WGP placement for matched native
+ experiments
+
+---
+ llvm_zluda/src/compile.rs | 42 ++++++++++++++++++++++++++++++++++-----
+ ptx/src/pass/llvm/emit.rs |  3 ++-
+ 2 files changed, 39 insertions(+), 6 deletions(-)
+
+diff --git a/llvm_zluda/src/compile.rs b/llvm_zluda/src/compile.rs
+index 90c0034..6a2f452 100644
+--- a/llvm_zluda/src/compile.rs
++++ b/llvm_zluda/src/compile.rs
+@@ -24,6 +24,37 @@ const OCKL_MODULE: &[u8] = include_bytes!("device-libs/ockl.bc");
+ // https://llvm.org/docs/AMDGPUUsage.html#address-spaces
+ const CONSTANT_ADDRESS_SPACE: u32 = 4;
+ 
++/// Opt-in RDNA scheduling experiment; changes placement, not FP semantics.
++pub fn wgp_requested() -> bool {
++    std::env::var("D4R_ZLUDA_WGP").is_ok_and(|value| value.trim() == "1")
++}
++
++fn target_features(wave64: bool, wgp: bool) -> &'static CStr {
++    match (wave64, wgp) {
++        (false, false) => c"-wavefrontsize64,+cumode",
++        (true, false) => c"-wavefrontsize32,+wavefrontsize64,+cumode",
++        (false, true) => c"-wavefrontsize64,-cumode",
++        (true, true) => c"-wavefrontsize32,+wavefrontsize64,-cumode",
++    }
++}
++
++#[cfg(test)]
++mod shader_mode_tests {
++    use super::target_features;
++
++    #[test]
++    fn wgp_changes_only_cu_placement_in_both_wave_modes() {
++        for wave64 in [false, true] {
++            let cu = target_features(wave64, false).to_str().unwrap();
++            let wgp = target_features(wave64, true).to_str().unwrap();
++            assert_eq!(cu.replace("+cumode", "-cumode"), wgp);
++            assert_eq!(cu.contains("+wavefrontsize64"), wave64);
++            assert_eq!(wgp.contains("+wavefrontsize64"), wave64);
++        }
++        assert_eq!(target_features(false, false), c"-wavefrontsize64,+cumode");
++    }
++}
++
+ fn load_module(ctx: &Context, bc: &[u8], name: &std::ffi::CStr) -> Result<Module, String> {
+     let module = Module::try_from_bitcode(ctx, bc, Some(name))
+         .ok_or(("Failed to parse bitcode").to_string())?;
+@@ -229,11 +260,12 @@ pub fn compile(
+     let cpu = std::ffi::CString::new(gcn_arch).map_err(|_| ("invalid gcn_arch").to_string())?;
+     // functions without their own target features (the linked helper libraries) take these, so under
+     // D4R_ZLUDA_WAVE64 they must be optimised as wave64 too (lane-id and ballot folding)
+-    let features = if std::env::var("D4R_ZLUDA_WAVE64").map_or(false, |v| v.trim() == "1") {
+-        c"-wavefrontsize32,+wavefrontsize64,+cumode"
+-    } else {
+-        c"-wavefrontsize64,+cumode"
+-    };
++    let wgp = wgp_requested();
++    if wgp && !gcn_arch.starts_with("gfx11") && !gcn_arch.starts_with("gfx12") {
++        return Err(format!("D4R_ZLUDA_WGP requires RDNA3/4, got {gcn_arch}"));
++    }
++    let features = target_features(
++        std::env::var("D4R_ZLUDA_WAVE64").is_ok_and(|v| v.trim() == "1"), wgp);
+ 
+     let mut target = unsafe { std::mem::zeroed() };
+     let mut err = std::ptr::null_mut();
+diff --git a/ptx/src/pass/llvm/emit.rs b/ptx/src/pass/llvm/emit.rs
+index 6af2495..8b67c93 100644
+--- a/ptx/src/pass/llvm/emit.rs
++++ b/ptx/src/pass/llvm/emit.rs
+@@ -397,12 +397,13 @@ impl<'a, 'input> ModuleEmitContext<'a, 'input> {
+ 
+     fn emit_target_features(&mut self, fn_: LLVMValueRef) {
+         let value = format!(
+-            "{},+cumode{}",
++            "{},{}{}",
+             if crate::pass::wave64_requested() {
+                 "-wavefrontsize32,+wavefrontsize64"
+             } else {
+                 "+wavefrontsize32,-wavefrontsize64"
+             },
++            if llvm_zluda::compile::wgp_requested() { "-cumode" } else { "+cumode" },
+             if cfg!(debug_assertions) {
+                 ",+precise-memory"
+             } else {
+-- 
+2.55.0.windows.5
+

scripts/windows/build-native-texture.ps1

@@ -4,6 +4,7 @@ param(
     [string]$Kernel = 'hiluma_engine_output_depthinv_mvlo_hdr_max_v2_rel',
     [string]$HipRoot = 'C:\Program Files\AMD\ROCm\7.2',
     [string]$ZludaRoot,
+    [ValidateSet('cu','wgp')][string]$ShaderMode='cu',
     [string]$OutputDirectory
 )
 $ErrorActionPreference='Stop'
@@ -11,5 +12,5 @@ $repo=[IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../..'))
 if (!$ZludaRoot) { $ZludaRoot=Join-Path $repo 'dist/zluda-windows-final' }
 if (!$OutputDirectory) { $OutputDirectory=Join-Path $repo 'build/private-textures-gfx1201' }
 & python (Join-Path $repo 'tools/windows/build_native_texture.py') --dlss-dll $DlssDll --hip-root $HipRoot `
-    --zluda-root $ZludaRoot --kernel $Kernel --output-directory $OutputDirectory
+    --zluda-root $ZludaRoot --kernel $Kernel --output-directory $OutputDirectory --shader-mode $ShaderMode
 if ($LASTEXITCODE) { throw "Native texture build failed ($LASTEXITCODE); inspect the output work directory." }

scripts/windows/build-zluda-windows.ps1

@@ -1,5 +1,5 @@
 [CmdletBinding()]
-param([string]$SourceRoot, [string]$LlvmBuildRoot, [string]$InstallRoot, [switch]$FetchOnly, [switch]$CacheTestsOnly, [int]$Jobs = 4)
+param([string]$SourceRoot, [string]$LlvmBuildRoot, [string]$InstallRoot, [switch]$FetchOnly, [switch]$CacheTestsOnly, [switch]$CompilerTestsOnly, [int]$Jobs = 4)
 $ErrorActionPreference = 'Stop'
 $repo = [IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../..'))
 if (!$SourceRoot) { $SourceRoot = Join-Path $repo 'external/ZLUDA' }
@@ -33,6 +33,14 @@ Push-Location $SourceRoot
 try {
     $cargo = Join-Path $rust 'bin/cargo.exe'
     if (!(Test-Path $cargo)) { throw 'Run setup-rust-toolchain.ps1 first.' }
+    if ($CompilerTestsOnly) {
+        $ErrorActionPreference = 'Continue'
+        & $cargo test --locked --release --target x86_64-pc-windows-gnu -p llvm_zluda shader_mode_tests `
+            2>&1 | Tee-Object (Join-Path $target 'compiler-tests.log')
+        $ErrorActionPreference = 'Stop'
+        if ($LASTEXITCODE) { throw "ZLUDA compiler regression tests failed ($LASTEXITCODE)" }
+        return
+    }
     if ($CacheTestsOnly) {
         $ErrorActionPreference = 'Continue'
         & $cargo test --locked --release --target x86_64-pc-windows-gnu -p zluda_cache `

scripts/windows/test-native-texture.ps1

@@ -1,8 +1,11 @@
 [CmdletBinding()]
-param([string]$TextureRoot, [string]$OutputDirectory, [int]$Iterations=8, [string]$OutputResolution='512x288')
+param([string]$TextureRoot, [string]$ControlTextureRoot, [string]$ZludaRoot,
+    [string]$OutputDirectory, [int]$Iterations=8, [string]$OutputResolution='512x288', [string]$InputResolution,
+    [switch]$CandidateFirst)
 $ErrorActionPreference='Stop'
 $repo=[IO.Path]::GetFullPath((Join-Path $PSScriptRoot '../..'))
 if (!$TextureRoot) { $TextureRoot=Join-Path $repo 'build/private-textures-gfx1201' }
+if (!$ZludaRoot) { $ZludaRoot=Join-Path $repo 'dist/zluda-windows-final' }
 if (!$OutputDirectory) { $OutputDirectory=Join-Path $repo 'test-results/k-native-texture-reference' }
 if ($OutputResolution -notmatch '^([0-9]+)x([0-9]+)$') { throw 'OutputResolution must be WIDTHxHEIGHT.' }
 $outputWidth=[int]$Matches[1]; $outputHeight=[int]$Matches[2]
@@ -13,19 +16,34 @@ foreach ($directory in @((Join-Path $repo 'build/native-k-gfx1201'),$TextureRoot
 }
 & python (Join-Path $repo 'kernels/tools/kernel_manifest.py') $combined (Join-Path $repo 'nvngx_dlss.dll')
 if ($LASTEXITCODE) { throw 'Combined private manifest failed.' }
+$controlNative=Join-Path $repo 'build/native-k-gfx1201'
+if ($ControlTextureRoot) {
+    $controlNative=Join-Path $OutputDirectory 'control-private-native'
+    New-Item -ItemType Directory -Force $controlNative | Out-Null
+    foreach ($directory in @((Join-Path $repo 'build/native-k-gfx1201'),$ControlTextureRoot)) {
+        Get-ChildItem -LiteralPath $directory -Filter '*.hsaco' | Copy-Item -Destination $controlNative -Force
+    }
+    & python (Join-Path $repo 'kernels/tools/kernel_manifest.py') $controlNative (Join-Path $repo 'nvngx_dlss.dll')
+    if ($LASTEXITCODE) { throw 'Control private manifest failed.' }
+}
 $settings=@{D4R_VALIDATE_OUTPUT='1'; D4R_PROFILE_STAGES='1'; D4R_ZLUDA_PROFILE='1'; D4R_QUIET_API='1';
+    D4R_ZLUDA_WMMA='1'; D4R_ZLUDA_WMMA_FP8='1'; D4R_ZLUDA_WMMA_FP8_NATIVE='0';
+    D4R_ZLUDA_WMMA_F16_REFERENCE='1'; D4R_ZLUDA_PROFILE_DEFERRED=$null;
     ZLUDA_CACHE_DIR=(Join-Path $repo 'build/zluda-cache-windows'); PYTHONPATH=(Join-Path $repo '.tools/python/vendor');
     D4R_ZLUDA_NATIVE_DIR=$null}
 $old=@{}
 try {
     foreach ($key in $settings.Keys) { $old[$key]=[Environment]::GetEnvironmentVariable($key,'Process'); [Environment]::SetEnvironmentVariable($key,$settings[$key],'Process') }
     foreach ($flags in @(0,11)) {
-        foreach ($mode in @('control','candidate')) {
+        $kernel=if ($flags -eq 11) { 'hiluma_engine_output_depthinv_mvlo_hdr_max_v2_rel' } else { 'hiluma_engine_output_depthreg_mvhi_ldr_max_v2_rel' }
-            $env:D4R_ZLUDA_NATIVE_DIR=if ($mode -eq 'control') { Join-Path $repo 'build/native-k-gfx1201' } else { $combined }
+        $order=if ($CandidateFirst) { @('candidate','control') } else { @('control','candidate') }
-            $arguments=@{RuntimeProfile='therock'; ZludaRoot=(Join-Path $repo 'dist/zluda-windows-final');
+        foreach ($mode in $order) {
+            $env:D4R_ZLUDA_NATIVE_DIR=if ($mode -eq 'control') { $controlNative } else { $combined }
+            $arguments=@{RuntimeProfile='therock'; ZludaRoot=$ZludaRoot;
                 PackageRoot=(Join-Path $repo 'dist/windows-rdna4-command-list'); NgxCore=(Join-Path $repo '_nvngx.dll');
                 DlssDll=(Join-Path $repo 'nvngx_dlss.dll'); NgxMode='d3d12'; NgxAbi='project-legacy'; Preset=11;
                 NgxCreateFlags=$flags; NgxOutputResolution=$OutputResolution; NgxOnly=$true; RequireNativeNetwork=$true; CommandListBackend=$true;
+                NgxInputResolution=$InputResolution;
                 PixelProfile='depth-stencil'; BarrierMode='inherited-legacy'; CaptureExceptions=$true; EarlyIndirectProbe=$true;
                 Iterations=$Iterations; TimeoutSeconds=300; OptiScalerDll=(Join-Path $repo 'dist/optiscaler-windows-d4r/OptiScaler.dll');
                 OutputDirectory=(Join-Path $OutputDirectory "$mode-$flags")}
@@ -33,9 +51,13 @@ try {
             if ($LASTEXITCODE) { throw "Texture $mode flags=$flags workload failed." }
             & python (Join-Path $repo 'tools/windows/profile_report.py') $arguments.OutputDirectory --output (Join-Path $arguments.OutputDirectory 'profile.json')
             if ($LASTEXITCODE) { throw 'Texture profile report failed.' }
+            $profile=Get-Content -LiteralPath (Join-Path $arguments.OutputDirectory 'profile.json') -Raw | ConvertFrom-Json
+            $samples=@($profile.kernels | Where-Object { $_.kernel -eq $kernel -and $_.backend -eq 'native' -and $_.phase -eq 'main' -and $_.serializing })
+            if ($samples.Count -ne 1 -or $samples[0].gpuMs.samples -ne $Iterations) {
+                throw "Missing valid serializing output-kernel timings for $mode flags=$flags; retain this run as correctness evidence only."
+            }
         }
         $control=Join-Path $OutputDirectory "control-$flags"; $candidate=Join-Path $OutputDirectory "candidate-$flags"
-        $kernel=if ($flags -eq 11) { 'hiluma_engine_output_depthinv_mvlo_hdr_max_v2_rel' } else { 'hiluma_engine_output_depthreg_mvhi_ldr_max_v2_rel' }
         $stderr=Get-Content -LiteralPath (Join-Path $candidate 'd3d12-evaluate-preset-11.stderr.log') -Raw
         $hits=[regex]::Matches($stderr, ('\[d4r-launch\] kernel="' + $kernel + '" backend=native')).Count
         if ($hits -ne $Iterations) { throw "Texture override was not executed on every frame: $kernel hits=$hits" }
@@ -43,6 +65,9 @@ try {
         if ($LASTEXITCODE) { throw "Texture flags=$flags full-frame RGB mismatch; preserve baseline." }
     }
     @{passed=$true; architecture='gfx1201'; framesPerVariant=$Iterations; outputResolution=$OutputResolution; createFlags=@(0,11); strictRgb=$true;
+        inputResolution=$InputResolution;
+        serializingProfile=$true; candidateFirst=[bool]$CandidateFirst;
+        controlTextureRoot=$ControlTextureRoot;
         manifestSha256=(Get-FileHash -LiteralPath (Join-Path $TextureRoot 'd4r-kernels.txt') -Algorithm SHA256).Hash.ToLowerInvariant();
         privateNvidiaDerivedCode=$true; source=(Get-Content -LiteralPath (Join-Path $TextureRoot 'build-info.json') -Raw | ConvertFrom-Json)} |
         ConvertTo-Json -Depth 12 | Set-Content -LiteralPath (Join-Path $TextureRoot 'validation.json') -Encoding UTF8

tools/windows/build_native_texture.py

@@ -21,6 +21,7 @@ def main():
     parser.add_argument('--zluda-root', type=pathlib.Path, required=True)
     parser.add_argument('--kernel', required=True)
     parser.add_argument('--output-directory', type=pathlib.Path, required=True)
+    parser.add_argument('--shader-mode', choices=['cu', 'wgp'], default='cu')
     args = parser.parse_args()
     if not re.fullmatch(r'hiluma_engine_output_depth(?:inv|reg)_mv(?:hi|lo)_(?:hdr|ldr)(?:_max)?_v2_rel', args.kernel):
         parser.error('this initial validated build recipe supports K v2 output variants only')
@@ -39,6 +40,7 @@ def main():
     env = os.environ.copy()
     env.update(D4R_PREFER_ACCURACY='1', D4R_ZLUDA_WMMA='1', D4R_ZLUDA_WMMA_FP8='1',
                D4R_ZLUDA_WMMA_FP8_NATIVE='0', D4R_ZLUDA_WMMA_F16_REFERENCE='1',
+               D4R_ZLUDA_WGP='1' if args.shader_mode == 'wgp' else '0',
                D4R_ZLUDA_IMPLICIT_MAX_BLOCK='256')
     for name in ('D4R_ZLUDA_IGNORE_DENORMAL', 'D4R_ZLUDA_FAST_MATH', 'D4R_ZLUDA_WMMA_F32ACC',
                  'D4R_ZLUDA_WAVE64', 'D4R_ZLUDA_PROFILE'):
@@ -74,9 +76,17 @@ def main():
     run('assemble', [tools['llvm-as'], ir, '-o', extra])
     env['D4R_ZLUDA_EXTRA_BC'] = str(extra)
     run('emit', [emitter, edited, work / 'emitted', 'gfx1201'])
+    optimized = (work / 'emitted/opt.ll').read_text()
+    definition = re.search(r'^define amdgpu_kernel .*@' + re.escape(args.kernel) + r'\(.*#(\d+)\s*\{', optimized, re.M)
+    if not definition:
+        raise RuntimeError('Cannot verify shader mode: kernel attributes are missing')
+    attributes = re.search(r'^attributes #' + definition[1] + r' = \{([^\n]+)\}', optimized, re.M)
+    expected_mode = '-cumode' if args.shader_mode == 'wgp' else '+cumode'
+    if not attributes or expected_mode not in attributes[1]:
+        raise RuntimeError(f'Emitter did not honor {args.shader_mode} mode; build the corresponding ZLUDA patch first')
     shutil.copyfile(work / 'emitted/module.hsaco', output / (args.kernel + '.hsaco'))
     run('manifest', [sys.executable, repo / 'kernels/tools/kernel_manifest.py', output, args.dlss_dll])
-    metadata = dict(architecture='gfx1201', accuracy=True, nativeFP8=False,
+    metadata = dict(architecture='gfx1201', accuracy=True, nativeFP8=False, shaderMode=args.shader_mode,
                     privateNvidiaDerivedCode=True, validation='not yet validated; do not install before comparison',
                     dlssSha256=hashlib.sha256(args.dlss_dll.read_bytes()).hexdigest(),
                     zludaBuild=json.loads((zluda / 'build-info.json').read_text(encoding='utf-8-sig')),