Merge upstream RDNA4 layouts and validate native Windows K integration

docs/building.md

@@ -4,7 +4,7 @@ These steps produce the pieces the launcher needs: a patched ZLUDA, a patched vk
 
 | Component | Tested version |
 |---|---|
-| ZLUDA | `ee2f25a` (upstream), plus `patches/zluda/0002`, `0003`, `0004`, `0005` in that order |
+| ZLUDA | `ee2f25a` (upstream), plus `patches/zluda/0002` through `0007` in order |
 | vkd3d-proton | `3dfc6f07` (the base GE-Proton11-3 ships), plus `patches/vkd3d-proton/0001` |
 | ROCm | 7.2 (HIP runtime, clang, device libraries) |
 | Proton | GE-Proton11-3 (with its OptiScaler integration) |
@@ -17,10 +17,11 @@ git clone https://github.com/vosen/ZLUDA zluda && cd zluda
 git checkout ee2f25a
 git submodule update --init --recursive
 git lfs pull
-for p in 0002 0003 0004 0005; do git apply /path/to/d4r/patches/zluda/$p-*.patch; done
+for p in 0002 0003 0004 0005 0006 0007; do git apply /path/to/d4r/patches/zluda/$p-*.patch; done
 # rebuild the device helpers the patches changed (ptx/lib/zluda_ptx_impl*.bc)
 ZLUDA_SOURCE_ROOT=$PWD ROCM_ROOT=/opt/rocm /path/to/d4r/scripts/build_zluda_ptx_helpers.sh
 LIBRARY_PATH=/opt/rocm/lib cargo build --release -p zluda
+LIBRARY_PATH=/opt/rocm/lib cargo build --release -p ptx --example d4r_emit
 mkdir -p ~/.cache/d4r-zluda-current
 cp target/release/libnvcuda.so ~/.cache/d4r-zluda-current/
 ln -sf libnvcuda.so ~/.cache/d4r-zluda-current/libcuda.so
@@ -34,6 +35,10 @@ What the patches add:
 - **0003**: an implicit 256-thread launch bound for kernels without PTX bounds (removes massive register spilling, `D4R_ZLUDA_IMPLICIT_MAX_BLOCK`) and f16 tensor-core MMA on RDNA3 WMMA (`D4R_ZLUDA_WMMA`).
 - **0004**: the native kernel override hook (`D4R_ZLUDA_NATIVE_DIR`, see [native-kernels.md](native-kernels.md)), FP8 MMA on WMMA (`D4R_ZLUDA_WMMA_FP8`), optional elision of per-instruction denormal mode switches (`D4R_ZLUDA_IGNORE_DENORMAL`), inlined image helpers, and linking extra bitcode into a PTX module (`D4R_ZLUDA_EXTRA_BC`, used by the texture-kernel build).
 - **0005**: a null texture object (CUDA handle 0) reads as zeros, as on NVIDIA GPUs, instead of faulting the GPU. DLSS samples absent optional inputs that way in some configurations (low-resolution motion vectors without HDR, as in Ghost of Tsushima).
+- **0006**: `m16n8k8` f16 MMAs (DLSS 3 CNN, presets E/F) on RDNA3 WMMA like the k16 ones (`D4R_ZLUDA_WMMA_K8=0` disables it); weight-image slots for native prep kernels (`d4r_prep_key_at`, `d4r_prep_key_slots`); and wave64 compilation for offline texture-kernel builds (`D4R_ZLUDA_WAVE64=1`, with wave64 builds of the helper bitcode). `scripts/build_zluda_ptx_helpers.sh` now writes the `_w64` helper variants too.
+- **0007**: gfx12 WMMA layout lowering, optional native e4m3 FP8 WMMA (`D4R_ZLUDA_WMMA_FP8_NATIVE=1`), and an architecture argument for `d4r_emit`. `D4R_ZLUDA_WMMA_LAYOUT=12` on gfx11 is a validation shim, not a release setting.
+
+When linking on a system with ROCm libraries outside the default search path, include their library directory in `LIBRARY_PATH`. The build also needs the appropriate ROCm link libraries. `CARGO_BUILD_JOBS=8` caps parallel Rust compilation if memory is limited.
 
 ## 2. vkd3d-proton
 
@@ -65,11 +70,11 @@ This stages the shim, the bridge, the NGX core and the DLSS feature library in `
 ```sh
 D4R_ROCM_DIR=/opt/rocm \
 D4R_DLSS_DLL=/path/to/nvngx_dlss.dll \
-D4R_ZLUDA_BUILD=~/.cache/d4r-zluda-current \
+D4R_ZLUDA_EMIT=/path/to/zluda/target/release/examples/d4r_emit \
 kernels/build.sh all kernels/out/native
 ```
 
-`kernels/out/native` then holds one code object per replaced DLSS kernel; set `NativeKernelDirFast` in d4r.ini (or `D4R_ZLUDA_NATIVE_DIR`) to it. `kernels/build.sh k` or `m` builds only the network layers and needs neither the DLL nor ZLUDA. The texture kernels (`tex`) extract PTX from your DLL into `kernels/extracted/`; that directory and the build output are git-ignored and must not be redistributed.
+`kernels/out/native` then holds one code object per replaced DLSS kernel; set `NativeKernelDirFast` in d4r.ini (or `D4R_ZLUDA_NATIVE_DIR`) to it. `kernels/build.sh k` or `m` builds only the network layers and needs neither the DLL nor ZLUDA. The texture kernels (`tex`) extract PTX from your DLL into `kernels/extracted/` and compile it offline for `D4R_GPU_ARCH` through `D4R_ZLUDA_EMIT`; that directory and the build output are git-ignored and must not be redistributed. On gfx12, build a second variant with `D4R_NATIVE_FP8=1` and place it in an `<arch>-fp8` folder.
 
 ## 6. Configure and play
 
@@ -93,14 +98,16 @@ scripts/package_release.sh            # -> dist/d4r-<version>.zip
 ```
 
 The script:
-- builds the shim, the bridge and the network-layer kernels;
+- builds the shim, the bridge and the network-layer kernels for gfx1100–gfx1103 and gfx1200–gfx1201 by default (`D4R_GPU_ARCHS` can select fewer targets); each gfx12 target also gets an `<arch>-fp8` variant;
 - writes the kernel manifest from the DLLs you list (it records hashes of their PTX, nothing else);
 - stages OptiScaler as `dxgi.dll` with the settings in `packaging/optiscaler.settings`;
-- adds NVIDIA's two DLLs and the texture kernels (built from NVIDIA's PTX by `kernels/build.sh tex`);
+- adds NVIDIA's two DLLs and any supplied texture kernels (built from NVIDIA's PTX by `kernels/build.sh tex`);
 - zips the result together with the ZLUDA and vkd3d-proton builds, the ROCm runtime (as `d4r/rocm`), `packaging/d4r.ini`, the licenses and the patches.
 
 The NVIDIA files are not covered by d4r's license; redistributing them is up to whoever publishes the zip. `D4R_BUNDLE_NVIDIA=0` builds `d4r-<version>-nonvidia.zip` without them and without the texture kernels. [architecture.md](architecture.md#portable-installs-the-release-zip) describes how the installed files work together.
 
+`D4R_BUNDLE_TEX` accepts the older flat directory for gfx1101, or a directory with per-target subdirectories (`gfx1100/`–`gfx1103/`, `gfx1200/`, `gfx1201/`, plus `gfx1200-fp8/` and `gfx1201-fp8/`). Build each texture set with `D4R_GPU_ARCH` and `D4R_ZLUDA_EMIT`; the `-fp8` sets also need `D4R_NATIVE_FP8=1`. `d4r_emit` targets those GPUs offline. Missing texture kernels fall back to ZLUDA on that target; the network-layer kernels are still included. Only the RX 7700 XT has been tested on real hardware.
+
 ## Checks
 
 - `scripts/check_environment.sh` lists the tools, GPU and Proton builds it finds.

docs/native-kernels.md

@@ -1,6 +1,6 @@
 # Native kernels
 
-The native kernels are RDNA3 (gfx11) code objects that ZLUDA runs in place of specific DLSS kernels. Each one has the same name, launch shape and parameter block as the PTX kernel it replaces, and reads the same buffers, including the network weights inside NVIDIA's DLL. No weights or NVIDIA code are stored here.
+The native kernels are RDNA3 (gfx11) and RDNA4 (gfx12) code objects that ZLUDA runs in place of specific DLSS kernels. Each one has the same name, launch shape and parameter block as the PTX kernel it replaces, and reads the same buffers, including the network weights inside NVIDIA's DLL. No weights or NVIDIA code are stored here.
 
 ## The override hook (patches/zluda/0004)
 
@@ -10,7 +10,9 @@ With `D4R_ZLUDA_NATIVE_DIR=DIR`, whenever NGX loads a PTX module and asks for ke
 |---|---|
 | `NAME_prep` kernel | launched first on the same stream, with the same parameters (typically re-lays out weights) |
 | `u32 d4r_prep_blocks` | grid size of the prep kernel (128 threads per block) |
-| `u32 d4r_prep_key_offset` | byte offset of a u64 in the parameter block; the prep is skipped while that value (the weights pointer) repeats |
+| `u32 d4r_prep_key_offset` | byte offset of a u64 in the parameter block; the prep is skipped while that value (the weights pointer) repeats (0 = no key) |
+| `u32 d4r_prep_key_at` | the same as offset + 1, so a key at offset 0 can be named (M's layers keep their weights pointer there) |
+| `u32 d4r_prep_key_slots` | N > 0: the kernel keeps one prepared image per key for up to N keys, so the prep runs once per weight set (the six enc3 tube blocks); keys beyond N share an overflow image and are prepared on every launch |
 | `u32 d4r_block_z` | replaces the launch's block z dimension (more waves per window) |
 | `u32 d4r_grid_x` | replaces the grid (persistent kernels) |
 
@@ -19,7 +21,7 @@ Kernels without a file in `DIR` are compiled from PTX as usual, so a partial set
 **Release directories.** A directory that holds `d4r-kernels.txt` (or `<gfx target>/d4r-kernels.txt`) is checked before use:
 - `kernels/tools/kernel_manifest.py DIR nvngx_dlss.dll...` writes the manifest. For each `NAME.hsaco` it records the FNV-1a 64 hash of the PTX module that defines `.entry NAME`, taken from each DLL given.
 - The CUDA bridge serves a kernel only after NGX has loaded a module with one of the listed hashes.
-- It picks the target folder from the GPU's KFD topology entry.
+- It picks the KFD GPU with the most SIMDs, then selects that GPU's target folder. `D4R_GPU_ARCH` overrides the choice. On gfx12, `D4R_ZLUDA_WMMA_FP8_NATIVE=1` selects the `<arch>-fp8` folder.
 
 The release is built this way, with hashes from DLSS 310.7.0 and 310.9.1. The PTX of every replaced kernel is identical in those two versions.
 
@@ -34,8 +36,20 @@ The release is built this way, with hashes from DLSS 310.7.0 and 310.9.1. The PT
 - **Weights:** prep kernels expand the weights into WMMA operand images once.
 
 **DLSS 4.5, preset M (`kernels/m`, `rrlite_*`).**
-- **Network:** the Swin blocks of DLSS 4.5, whose weights are FP8. Their prep kernels expand the weights to f16 WMMA operands.
+- **Network:** the Swin blocks of DLSS 4.5, whose weights are FP8. Their prep kernels expand the weights to f16 WMMA operands, once per weight set.
 - **Template:** `swin_block.h` covers encoders, the tube-shaped enc3 and decoders.
+- **Output encoding:** the FP8 output is encoded two values at a time with packed 16-bit operations (`enc8x2`, exhaustively equal to the scalar encoder), and the 2×2 patch merge reads the rounded f16 values the codes decode to, from a row layout without LDS bank conflicts, instead of decoding the bytes again in every wave.
+- **Weight loads:** enc1 forms each weight tile's address in scalar registers (`SWIN_SCALAR_BLOAD`); the 8-wave layers are faster without it.
+
+## RDNA4
+
+`kernels/common/wmma_layout.h` selects the WMMA operand and accumulator layout. gfx12 uses eight f16 values per lane in each K half; the accumulator rows are `i + 8·half`. The gfx11 path retains its original layout. `D4R_WMMA_LAYOUT=12` on gfx11 is a test shim: it exercises gfx12 indexing but executes gfx11 WMMA, so its replay output can be compared byte for byte with the existing gfx11 build.
+
+The gfx12 M variant can use native e4m3 FP8 WMMA with `D4R_NATIVE_FP8=1`. Its activations are requantized to e4m3 at operand load, and its prepared weights remain bytes. The matching ZLUDA and texture-tail variant uses `D4R_ZLUDA_WMMA_FP8_NATIVE=1` / `D4R_TEX_FP8=1`; the release bridge chooses the `-fp8` kernel folder. `NativeFp8` in d4r.ini controls that choice and defaults to on. The f16 widening path remains available.
+
+The gfx11 refactor and gfx12-layout shim matched the recorded gfx11 replays and 88-frame K/M harness images byte for byte on the RX 7700 XT. rocjitsu gfx1201 replay comparisons put the native K layers within 96–109 dB PSNR of gfx11, and M differed in 0.001–0.3% of e4m3 bytes. These are emulator results, not RDNA4 hardware results. rocjitsu's FP8 arithmetic is not hardware-checked; bit-exactness, speed and driver stability on an RDNA4 GPU remain unverified.
+
+For ZLUDA-compiled code, compare rocjitsu runs with other rocjitsu runs: the emulator does not reproduce the denormal-flush mode of those code objects, so a GPU byte comparison is not a valid arithmetic check. Captures with texture or surface objects must not be sent to the replay tools; the standalone `tail_check.hip` and the full D3D12 harness cover those tails.
 
 **Texture kernels (`kernels/tex`).**
 - **The problem:** some DLSS kernels are mostly texture sampling and scalar maths that ZLUDA already compiles well, apart from a few slow parts.
@@ -43,8 +57,10 @@ The release is built this way, with hashes from DLSS 310.7.0 and 310.9.1. The PT
   - surface stores become calls into `tex_common.h`;
   - the `roundf` idiom that forces ZLUDA into strict-FP mode is rewritten exactly;
   - for M's enc0 and dec0, a range of the kernel is replaced by a native tail or head.
-- **Build:** ZLUDA compiles the edited PTX with the HIP bitcode linked in (`D4R_ZLUDA_EXTRA_BC`), and the resulting code object is saved.
-- **Output redirect:** the K output kernel (`hiluma_engine_output_*`) can also write its result straight into the shim's output buffer instead of a CUDA array. See `D4R_SHIM_OUTPUT_DIRECT` in [architecture.md](architecture.md).
+- **Build:** ZLUDA's `d4r_emit` compiles the edited PTX for `D4R_GPU_ARCH` with the HIP bitcode linked in (`D4R_ZLUDA_EXTRA_BC`), and the resulting code object is saved. This is an offline per-target build.
+- **Every flag combination:** DLSS picks kernel variants by the game's settings (motion vectors at render or display resolution, HDR or LDR input, inverted or regular depth, ...). `kernels/build.sh` builds all of them: 24 K output kernels, 8 M post, 4 M downsample and 4 M enc0 kernels. enc0's native tail is spliced into each variant at its own line, with the registers read off the variant's PTX (`make_ptx.py` checks that the code after the cut matches the reference variant line for line).
+- **Wave64:** kernels without MMAs can be compiled as wave64 (`D4R_ZLUDA_WAVE64=1`, ZLUDA patch 0006): each wave runs two CUDA warps, and RDNA3 issues FP32 work for all 64 lanes at once. The post and K output kernels are built this way (post 0.89 → 0.77 ms), with bit-identical results.
+- **Output redirect:** the kernel that writes a preset's result can write it straight into the shim's output buffer instead of a CUDA array: `hiluma_engine_output_*` for K, `rrlite_downsample_kernel_*` for M (M's post kernel works at 3840×2160; the downsample produces the output). See `D4R_SHIM_OUTPUT_DIRECT` in [architecture.md](architecture.md).
 
 ## Numerics
 
@@ -61,7 +77,7 @@ kernels/build.sh tex        # texture kernels (needs D4R_DLSS_DLL and D4R_ZLUDA_
 kernels/build.sh all DIR    # everything into DIR (default kernels/out/native)
 ```
 
-`D4R_GPU_ARCH` selects the target (default gfx1101). All kernels need gfx11 WMMA.
+`D4R_GPU_ARCH` selects one target for this standalone build (default gfx1101). The release script builds the network layers for gfx1100–gfx1103 and gfx1200–gfx1201 by default and puts each set in its own directory. The network kernels need gfx11 or gfx12 WMMA. Texture-kernel builds use `D4R_ZLUDA_EMIT` to generate code objects for the selected target without that GPU. Set `D4R_NATIVE_FP8=1` on gfx12 for the matching FP8 M and texture variants.
 
 ## Validating a kernel
 
@@ -72,23 +88,25 @@ kernels/build.sh all DIR    # everything into DIR (default kernels/out/native)
    PREP_GRID=<d4r_prep_blocks> dump_runner NAME.hsaco NAME <capture> <out> 30
    ```
 
-   It runs the prep and main kernels, saves every buffer after the first launch, and times 30 launches.
+   It runs the prep and main kernels, saves every buffer after the first launch, and times 30 launches. The replay tool refuses captures with texture or surface objects; use the full harness for those kernels.
 3. **Compare against a reference.** `kernels/tools/pwin_model.py` (K) and `swin_model.py` (M) are numpy models of the layers, written stage by stage against `ptxsim.py`, a small vectorised PTX interpreter that runs NVIDIA's kernels on the CPU. A bit-exact variant should match the model within f16 rounding; after any change, compare the new build's buffers with the previous build's.
 4. **Check the whole pipeline.** Replay captured frames through the harness with both kernel sets and compare the outputs with `kernels/tools/psnr.py`. `D4R_CUDA_KERNEL_PROFILE=1` makes the bridge time every kernel; `kernels/tools/kprof.py` summarises the log per frame.
 
 ## Per-kernel cost
 
-GPU time per frame at Quality (1705×960 → 2560×1440, RX 7700 XT), from `D4R_CUDA_KERNEL_PROFILE`:
+Median GPU time per frame at Quality (1706×960 → 2560×1440, RX 7700 XT), from `D4R_CUDA_KERNEL_PROFILE` in the D3D12 harness (flags as in Townfall; frames 21–40). The profiler's per-launch events add a little to every kernel, so totals are slightly above the in-game cost. "Before" is release 0.1.1.
 
-| DLSS 4 (K) | ms | DLSS 4.5 (M) | ms |
-|---|---|---|---|
-| hiluma output (texture kernel) | 0.82 | enc3 tube (6 launches) | 1.42 |
-| dec0 | 0.30 | enc1 | 1.27 |
-| enc0 | 0.28 | dec1 | 1.18 |
-| dec1 | 0.24 | enc2 | 0.79 |
-| hiluma input (ZLUDA) | 0.24 | post (texture kernel) | 0.77 |
-| enc1 | 0.21 | dec2 | 0.69 |
-| enc2, dec2 | 0.10 each | enc0 (texture kernel) | 0.60 |
-| enc3, dec3, enc4, dec4, dec5 | 0.05–0.08 each | dec0 (texture kernel) | 0.46 |
-| NGX exposure and misc | 0.1 | downsample, NGX misc | 0.36 |
-| **total** | **2.8** | **total** | **7.5** |
+| DLSS 4 (K) | before | now | DLSS 4.5 (M) | before | now |
+|---|---|---|---|---|---|
+| hiluma output (texture kernel) | 0.92 | 0.89 | enc3 tube (6 launches) | 1.62 | 1.57 |
+| dec0 | 0.33 | 0.32 | enc1 | 1.41 | 1.27 |
+| enc0 | 0.31 | 0.31 | dec1 | 1.44 | 1.43 |
+| dec1 | 0.26 | 0.26 | enc2 | 0.87 | 0.77 |
+| hiluma input (ZLUDA) | 0.25 | 0.24 | post (texture kernel) | 0.88 | 0.77 |
+| enc1 | 0.23 | 0.22 | dec2 | 0.84 | 0.82 |
+| enc2, dec2 | 0.12, 0.10 | 0.12, 0.10 | enc0 (texture kernel) | 0.64 | 0.63 |
+| enc3, dec3, enc4, dec4, dec5 | 0.07–0.09 each | 0.07–0.09 each | dec0 (texture kernel) | 0.64 | 0.63 |
+| NGX exposure and misc | 0.1 | 0.1 | downsample, NGX misc | 0.32 | 0.33 |
+| **total** | **3.10** | **2.97** | **total** | **9.04** | **8.23** |
+
+Runs of the same build vary by about ±3% per kernel. M additionally skips its output copy now (direct output through the downsample kernel), which is not a kernel and not in the table. Games whose flags differ from Townfall's (motion vectors at render resolution, LDR input, regular depth) gain more: those kernel variants used to run as plain ZLUDA compiles (K 3.6–4.6 ms → 2.8–3.0 ms, M −0.5 to −0.7 ms per frame).

docs/performance.md

@@ -1,5 +1,7 @@
 # Performance
 
+All frame rates and kernel timings on this page were measured on an RX 7700 XT (gfx1101). RDNA4 (gfx1200/gfx1201) has compile and emulator validation only; no RDNA4 frame rate, latency, or kernel timing has been measured. Native FP8 may change both arithmetic and cost, so the gfx11 results cannot be projected to it.
+
 ## Method
 
 - **Hardware and settings:** Radeon RX 7700 XT (RDNA3, 54 CUs), Ryzen 9 5900XT, Linux 6.18. SILENT HILL Townfall at 2560×1440 output.
@@ -8,18 +10,20 @@
 - **Pairing:** runs are compared back-to-back in one session, because repeated runs of an identical setup vary by about 1%.
 - **Screen recording:** it costs a few fps, so recorded videos show slightly lower numbers than the table.
 - **Latency:** all DLSS numbers are same-frame (frame age 0). Every frame shows its own DLSS result, as with a native upscaler.
-- **Other upscaler modes:** Balanced, Performance and Ultra Performance use OptiScaler's render-ratio override (1.72, 2.0, 3.0) on top of the game's Quality mode.
+- **Upscaler modes:** every mode uses OptiScaler's render-ratio override (1.5, 1.72, 2.0, 3.0), so the result does not depend on the DLSS mode chosen in the game.
 
 ## Results
 
-| Mode (render resolution) | DLSS 3 CNN (E) | DLSS 4 (K) | DLSS 4.5 (M) | FSR 4 |
+| Mode (render resolution) | DLSS 3 CNN (E) | DLSS 4 (K) | DLSS 4.5 (M) | FSR 4\* |
 |---|---|---|---|---|
-| Quality (1705×960) | **72.4** | 69.4 | 51.5 | 76.0 |
-| Balanced (1488×837) | **80.5** | 76.6 | 59.6 | 84.5 |
-| Performance (1280×720) | **87.8** | 84.0 | 69.9 | 94.0 |
-| Ultra Performance (853×480) | 88.9 | **94.5** | 90.3 | 107.3 |
+| Quality (1705×960) | **71.7** | 67.9 | 49.3 | 76.0 |
+| Balanced (1488×837) | **80.8** | 75.9 | 58.0 | 84.5 |
+| Performance (1280×720) | **89.7** | 84.1 | 69.0 | 94.0 |
+| Ultra Performance (853×480) | 89.1 | **95.5** | 92.2 | 107.3 |
 | Native 2560×1440, no upscaling (TSR at 100%) | | 49.1 | | |
 
+\* FSR 4 and the native row are from 2026-09-27; the DLSS columns from 2026-09-29, when the machine ran about 2–4% slower overall (release 0.1.1 at Quality that day: E 71.2, K 67.9, M 48.4 fps, against 72.4, 69.4 and 51.5 two days earlier). The render resolution of every mode is set with OptiScaler's ratio override for whichever DLSS mode the game is set to (ratios 1.5, 1.72, 2.0 and 3.0).
+
 DLSS 4's cost is almost constant across modes (about 2.8 ms of GPU time per frame). Its network runs on a grid set by the output resolution, not the render resolution, so the gap to FSR 4 widens as the render resolution drops. The CNN gets more expensive at the 3× ratio, so K is the better choice at Ultra Performance.
 
 ## What each step contributed

docs/windows-rdna4-port.md

@@ -18,7 +18,7 @@ Do not integrate NGX until the integer PTX workload is stable on the real GPU.
 | M2: Windows ZLUDA integer PTX, primary and created contexts | PASS, 32 iterations each + guard verification |
 | M3: Windows NGX initialization and simple DLSS path | Init + SR capabilities PASS; Evaluate/transformer pending |
 | M4: D3D12 / HIP external memory and fence round trip | PASS with TheRock; stable 7.2 has mapped-view leak |
-| M5: independently validated gfx12 WMMA backend | PASS: raw + legacy adapter, 16 cases, max abs/relative error 0 |
+| M5: independently validated gfx12 WMMA backend | PASS: raw + legacy adapter + upstream layout, max abs/relative error 0 |
 | M6: K layers, full transformer and image validation | Partial: `enc1`/`enc2` execute and match nonzero references; other layers/full network pending |
 | M7: M FP16-equivalent baseline and full transformer | Pending |
 | M8: standalone Windows D3D12 NGX harness | Pending |
@@ -50,6 +50,77 @@ Do not integrate NGX until the integer PTX workload is stable on the real GPU.
   This is a portable fallback on this machine, where MSVC is not installed.
   The device compiler remains the selected AMD HIP clang.
 
+## Upstream RDNA4 update (checked 2026-09-30)
+
+Fetched and integrated upstream main commit
+[`dbef4b24f4bc974725c3b2bc74ca442a694b901d`](https://github.com/countervolts/d4r/commit/dbef4b24f4bc974725c3b2bc74ca442a694b901d),
+"Add RDNA4 (gfx12) support and a 0.1.2 optimization round" (2026-09-30 UTC).
+It changes 42 files and supplies real per-target fragment layouts rather than
+renaming gfx11 builtins. Before the merge, an isolated archive was compiled
+and its `enc1`/`enc2` objects were checked on this RX 9070 XT against our fixtures.
+
+* `kernels/common/wmma_layout.h` separates gfx11/gfx12 operands, accumulator
+  rows, weight/activation loading, packing and reconstruction. K, M and native
+  texture tails use it. The gfx11 implementation and layout12 test shim remain.
+* M has an optional native e4m3 FP8 variant, with matching ZLUDA/texture lowering
+  and kernel directory selection. **Windows CMake keeps FP16 widening as the
+  baseline; it does not enable native FP8.** M arithmetic on this GPU remains
+  unvalidated until K and its full DLSS integration are correct.
+* New ZLUDA patches `0006`/`0007` add k8 WMMA, prep slots, wave64 for MMA-free
+  kernels, gfx12 helpers and FP8 selection/cache identity. These patches are
+  present in our branch but have not been built or validated in Windows ZLUDA.
+  Our current integer PTX test uses the unpatched `v7-preview.11` binary.
+* Upstream restored `kernels/tools/model_enc3.py`, resolving the missing M
+  reference dependency found in the initial audit. New check/replay utilities
+  cover K/M and texture tails. Texture artifacts still require local DLL/PTX
+  extraction and a patched `d4r_emit`; they are not built by Windows CMake yet.
+* There is still no native Windows runtime backend in this update. GPU selection
+  still reads KFD topology; Wine, Linux shared descriptors and vkd3d-proton
+  command-list splitting remain in the upstream path.
+
+Upstream explicitly reports gfx1201 **emulator** checks and gfx1200 compile-only
+coverage; it does not claim actual RDNA4 hardware verification
+([native-kernel validation notes](https://github.com/countervolts/d4r/blob/dbef4b24f4bc974725c3b2bc74ca442a694b901d/docs/native-kernels.md)).
+The hardware results below apply only to the named Windows probes and synthetic
+layers, not to the complete network or the upstream FP8 implementation.
+
+Local integration retains the Windows probes, NGX ABI correction, NVAPI topology
+and D3D12/HIP interop. K now uses upstream's native gfx12 layout directly; the
+legacy adapter remains only for its independent diagnostic regression. M gains
+the same minimal device-header path as K. Its lane-mask constants no longer
+require a host `<type_traits>` installation. Public HIP `uint2`/`uint4` storage
+alignment and qualifiers are reproduced by the device-only header.
+
+Windows CMake compiles all **11 K + 5 M** source modules with gfx1201 wave32,
+`-O3` and each source's upstream `d4r-build-flags` (including `-mcumode`). Both
+stable HIP 7.2 and isolated TheRock build successfully. The objects install to
+`experimental/k` and `experimental/m`, outside the ZLUDA override directory.
+Only `enc1`/`enc2` currently have GPU layer correctness checks.
+
+2026-09-30 integration results on RX 9070 XT, Windows 11:
+
+| Check | Result |
+| --- | --- |
+| TheRock diagnostic runner with the two local NVIDIA DLLs | 11/11 PASS; `test-results/upstream-rdna4-merged.zip` |
+| TheRock CTest, including D3D12/HIP external memory/fences | 8/8 PASS |
+| WMMA raw, legacy adapter and upstream native layout/packing | 32 cases, one/two K16 steps, nonzero C; max abs/relative error 0 |
+| Upstream K enc1 full / merged reference | Max abs `0.000244140625`; PSNR `98.21 / 97.63 dB`; no NaN/Inf |
+| Upstream K enc2 full / merged reference | Max abs `3.81469727e-06`; PSNR `130.39 / 132.49 dB`; no NaN/Inf |
+| Stable HIP 7.2 non-interop CTest | 6/6 PASS |
+| Stable-built K enc1/enc2 with stable runtime | 32 identity iterations each + both nonzero references PASS, same errors as TheRock |
+
+The previously failing stable compiler/legacy-layout K path is superseded by
+the native upstream layout: these two current layer fixtures pass under stable
+HIP 7.2. Stable external-memory mapped-view ownership still requires the runtime
+fix/TheRock; passing K does not resolve that leak. Build logs remain under each
+`build/windows-rdna4*` directory; stable GPU logs/fixtures remain under
+`test-results/upstream-stable-*-gpu.log` and `test-results/upstream-rdna4-stable`.
+
+Next gate: validate the remaining K layers against references/captures, then
+build the d4r-patched Windows ZLUDA and implement same-frame Windows NGX Evaluate
+with explicit D3D12 queue submission ownership. Full K precedes M and native
+FP8/performance work. No complete DLSS or OptiScaler game result is claimed.
+
 ## Architecture and platform boundaries
 
 Existing path:
@@ -122,31 +193,35 @@ or update all dependent loads/stores and intermediate fragments together.
 Start with FP16 / FP32 accumulate. M native FP8 is a later measured optimization.
 Compile success is not proof of lane layout or numerical correctness.
 
-`kernels/common/wmma_backend.h` now preserves the gfx11 call and maps d4r's
+The initial `kernels/common/wmma_backend.h` adapter preserves the gfx11 call and maps d4r's
 16-half operand / even-odd accumulator contract to gfx12's 8-half operand /
 contiguous accumulator contract. The gfx12 mapping uses an unconditional
 wave32 half-exchange for every source fragment element before selection.
 Performing the exchange only on lanes selected by a group-dependent branch
 returned channel 9 in place of channel 1 on this card: the opposite half of
 the wave was inactive at that instruction. This was caught by the CPU reference.
-`tools/windows/wmma_gfx1201.hip` tests raw gfx12 WMMA, the adapter, the
-half-exchange and lane ID with nonzero accumulator and one/two K16 steps.
+`tools/windows/wmma_gfx1201.hip` tests raw gfx12 WMMA, the adapter, upstream's
+native layout/operand packing, the half-exchange and lane ID with nonzero
+accumulator and one/two K16 steps.
 Both stable HIP 7.2 and TheRock 10.2 produced exact 16x16 outputs across 16
 different inputs; max absolute and relative errors were both zero.
 
-`kernels/k/pwin_common.h` now calls the tested adapter. The Windows device-only
+After the upstream merge, `kernels/k/pwin_common.h` calls the native
+`wmma_layout.h` implementation; it no longer converts around every MMA with
+the legacy adapter. The Windows device-only
 compiler uses `kernels/common/hip_device_minimal.h`, exposing only public
 Clang AMDGPU work-item/grid/barrier builtins and HIP-compatible qualifiers;
-the ordinary Linux HIP include path remains. CMake builds `enc1`/`enc2` for gfx1201
-into `experimental/k/` and verifies that HIP loads both prep and transformer
-entry points. The `k_module_probe` now launches prep and transformer. The
-identity fixture verifies all 4096 full-resolution and 1024 merged FP16 values
+the ordinary Linux HIP include path remains. CMake builds every K/M module for
+gfx1201 into `experimental/`. The `k_module_probe` resolves and launches prep
+and transformer for `enc1`/`enc2`. The
+identity fixture verifies all 4096 full-resolution and 1024/1536 merged FP16 values
 bitwise. A second fixture enables nonzero V projections, position-only
 attention, Wo, MLP/GELU and patch merge; it checks finite values and a patch
 identity, then saves all inputs/outputs for `k_layer_validate.py`. That script
 runs the existing `pwin_model.py` and measures max absolute/relative error and
 PSNR. The code object remains in `experimental/k/`, outside ZLUDA's override
 path, until the other K layers and real captured weights/activations pass.
+The enc2 fixture additionally enables learned Q/K attention and softmax.
 
 ## External interop blockers
 
@@ -361,19 +436,21 @@ Bundle `test-results/m6-k-enc2-final.zip`: 11/11 gates PASS. CTest: 8/8 PASS.
 The generic probe/reference runner selects `--kernel-name enc1|enc2`; both
 objects remain under `experimental/k` and are not used as DLSS overrides yet.
 
-Stable HIP SDK 7.2 compiler caveat: its `-O2` code object for `enc1` produces
+Historical stable HIP SDK 7.2 compiler caveat (pre-upstream legacy adapter):
+its `-O2` code object for `enc1` produces
 NaN already on the identity fixture (first element `0xfe00` instead of
 `0xbc00`). Its `-O1` object also miscomputes the identity fixture; `-O0`
 passes but is unsuitable for the fast path. Cross-testing isolates the problem
 to the stable compiler/code object: the stable-built `-O2` object fails under
 TheRock runtime, while the TheRock-built `-O2` object passes under stable HIP
-runtime. The independent WMMA probe passes under both compilers. Until a
-smaller compiler reproducer or source-level workaround is found, use the
-TheRock compiler for native K and the TheRock runtime for leak-free interop.
+runtime. The independent WMMA probe passes under both compilers. That legacy
+path required the TheRock compiler for native K. The 2026-09-30 native-layout merge
+supersedes this failing path: current enc1/enc2 fixtures pass with stable's
+`-O3` objects too. Keep TheRock runtime for leak-free interop.
 
-Audit found an upstream validation blocker: `swin_model.py` imports
-`model_enc3`, which is absent from the repository. M reference validation needs
-that dependency restored or an independently checked replacement.
+The initial audit found an upstream validation blocker: `swin_model.py` imported
+absent `model_enc3`. Upstream commit `dbef4b2` restores this file; the missing
+dependency is resolved, while real GPU M reference validation is still pending.
 
 Environment-specific issues resolved: sandbox disallows writes to `.git` and
 launching the MinGW child compiler, so branch/commit/build require scoped tool