kernels/tex: native token-lane parts for preset L's enc0 and dec0

L's unfolded enc0 and dec0 are still ZLUDA's compile of NVIDIA's PTX and cost
2.95 ms of the preset's 5.86 ms per frame (RX 9070 XT, 1280x720 -> 4K). The
embedding, the patch expand, the Swin block (32 channels, 2 heads, at pixel
resolution) and the patch merge are now native token-lane code; the texture
work around them stays ZLUDA's compile.

enc0l_tail.hip cuts in at the barrier after the feature staging and dec0l_block.hip
before the output code that reads the block's f16 result out of the last MMAs; make_ptx.py
splices either in when D4R_TEX_SRC names it and build.sh selects both for the gfx12
native-FP8 builds. swin_tex.h carries the indexing the two tails share; swin_tok.h gains
the pixel-order token layout (SWIN_TOK_NATURAL), ZLUDA's attention key order
(SWIN_PV_ZLUDA_ORDER) and swin_tok_core, which the encoded tests reuse. swin_common.h
takes the block weight layout (moved out of swin_block.h) and the exact and approximate
reciprocal / mix forms the tails need without the HIP headers (D4R_NO_HIP_RUNTIME).

The f32 sums stay in ZLUDA's order (K slots of the FP8 WMMAs, key order of the f16
attention product) and rsqrt/rcp use the hardware approximations ZLUDA lowers to, so the
harness's final frame is byte-identical to the translated kernels: all four enc0 flag
variants, 720p and 1080p render, 1440p and 4K output, static and moving scenes, 100
frames, and the accuracy set. With M's own sum order about one token in 20,000 differed.
M's layers compile to the same instructions and its frame is byte-identical.

enc0 1.69 -> 0.63 ms and dec0 1.26 -> 0.54 ms at 720p->4K, 2.73 -> 1.16 and 2.28 -> 1.01
at 1080p->4K; the preset 5.83 -> 4.06 and 8.63 -> 5.69 ms. -DD4R_SWIN_K32_MIX rounds the
f16 products earlier for 0.56 and 0.49 ms and is not byte-identical, so it is not built by
default. The repo's tests pass (36, 3 skipped).

docs/native-kernels.md

@@ -57,6 +57,16 @@ The release is built this way, with hashes from DLSS 310.7.0 and 310.9.1. The PT
 
 **Motion mitigation.** `[Kernels] NativeSwinEncoders = false` (`D4R_NATIVE_SWIN_ENCODERS=0`) bypasses only `rrlite_enc1_4x4` and `rrlite_enc2_4x4` in the native manifest. L/M then use the original translated PTX for those encoders, retaining native tube, decoder and texture kernels. Keep `PreferAccuracy = true` for the tested conservative path. It reduced the extra building-edge trails in a 64-frame moving Townfall capture at 853×480 → 2560×1440 on gfx1101; it does not recover detail absent from the low-resolution input or establish RTX parity. The default remains `true` for speed. Flat developer sets without manifests fall back entirely when this switch is off; add a manifest to keep selective acceleration.
 
+**L's enc0 and dec0 on RDNA4 native FP8 (`kernels/tex/enc0l_tail.hip`, `dec0l_block.hip`).** The unfolded kernels contain a whole Swin block at pixel resolution (32 channels, 2 heads, 4×4 windows) that M's folded variants do not have: enc0 is the feature code, a two-layer embedding (32 → 32 → 32 with a ReLU), the block, and the 2×2 patch merge to 64 channels; dec0 is the patch expand of the half-resolution tokens, the skip sum, the block, and output code that writes four surfaces. On the `-fp8` build `make_ptx.py` keeps the texture code (enc0's feature staging, dec0's output code) as ZLUDA's compile and replaces the rest with the token-lane block of `swin_tok.h` (`swin_tok_core`), one window after the other in the kernel's single wave per 8×8-pixel block.
+
+- **enc0:** the cut is the barrier after the feature staging; the tail reads the staged features from shared memory and loads the parameters and the block index itself, so no register of the original code is involved.
+- **dec0:** the output code reads the block's f16 result from the D registers of the last 16 MMAs and otherwise depends on nine trivial statements, which `make_ptx.py` finds by following its inputs backwards and keeps. The native function leaves the result in shared memory in that register layout and the edited PTX loads it back.
+- **Same bytes as ZLUDA's compile.** These kernels replace code that ZLUDA compiled, so they reproduce its f32 arithmetic where it is not exact (`swin_tex.h`): the two WMMAs of a k32 step hold K 0..15 and K 16..31 in order with C added in the first (`SWIN_TOK_NATURAL`, `K32_NO_MIX`), the f16 P V product has its keys in the K slots of ZLUDA's m16n8k16 lowering (`SWIN_PV_ZLUDA_ORDER`), and the reciprocal square root and reciprocal are `v_rsq_f32` / `v_rcp_f32` (`SWIN_APPROX_RECIP`). The order matters because f32 sums that are not exact round differently in a different order: with M's own slot order the P V product gave other codes than ZLUDA's compile in about one pixel token in 20,000 (each of the two orders then disagrees with `ptxsim.py`'s exact sums in different tokens), and with the single-rounding k32 form one merged code differed in the seventh 3840×2160 frame, where a k32 sum lay 0.0001 ulp from a rounding tie.
+- **Validation (RX 9070 XT):** `ptxsim.py` runs the original enc0 from the staging barrier on (texture code stubbed, staged features replaced) and a numpy model built on `swin_model.py` matches its stores on interior and border blocks; the harness's final frame is byte-identical to the `sust_only` builds for all four enc0 flag variants, 1280×720 and 1920×1080 render, 2560×1440 and 3840×2160 output, static and moving scenes, 100 frames, and in the accuracy set.
+- **Cost:** enc0 1.69 → 0.63 ms and dec0 1.26 → 0.54 ms at 1280×720 → 3840×2160 ([performance.md](performance.md#rdna4)). The inline assembly of `mix2` is not available in a ZLUDA-compiled kernel (its LLVM has no assembler); `-DD4R_SWIN_K32_MIX` (`D4R_TEX_CFLAGS`) selects the same instructions from `fma(P, one, c)` instead: enc0 0.63 → 0.56 ms and dec0 0.54 → 0.49 ms, but a near-tie sum then rounds as the exact sum does and not as ZLUDA's compile, so it is not byte-identical (first difference in the seventh frame of the test above, 40 dB after 30 frames of temporal feedback). It is not built by default.
+
+Other targets (RDNA3, and RDNA4 with `NativeFp8` off) keep the `sust_only` builds of these kernels.
+
 Initial validation: a gfx1101 fast L build completed two Ultra Performance harness frames (192×108 → 576×324, DLSS 310.7). Its finite, nonzero RGBA16F output matched the original translated L texture kernels byte for byte with the same native Swin layers. A gfx1201 FP8 accuracy L set compiled successfully; RDNA4 hardware and 4K game output have not been tested. Full packaging rebuilds missing L variants when given an older texture bundle.
 
 ## RDNA4
@@ -73,6 +83,7 @@ In that build the GEMM activations live in LDS as e4m3 bytes (`SWIN_A8` in `swin
 - The input is decoded with `v_cvt_pk_f32_fp8`. It equals `e4m3_to_half` for every code except the NaN codes `0x7f` / `0xff` (NaN instead of ±480); the FP8 WMMAs already read those bytes that way.
 - The MLP clamp uses the NaN-propagating minimum / maximum; it differs from minnum / maxnum only for a NaN input, whose product is NaN either way.
 - `mix2` (all exact FP8 layers, enc2 and dec2 included) rounds in place: the accumulator is both addend and destination, which removes a register copy per pair.
+- The part between a block's input and its output codes is `swin_tok_core`, which the L texture kernels below reuse. M's layers compile to the same instructions as before the split (a few inverted branches), and M's harness frames are byte-identical.
 - `-DSWIN_NO_TOKEN_LANES` builds these layers from `swin_block.h` again. enc2 and dec2 (96 channels, four heads) use the same template: byte-identical on the same recorded launches and harness sequences, 0.22 → 0.14 and 0.20 → 0.14 ms at 1280×720 render, 0.75 → 0.47 and 0.69 → 0.47 ms at 2560×1440.
 - The tube retains upstream's eight weight-image slots plus an overflow slot. Weights and per-channel constants use the same slot; `d4r_prep_identity_at` lets MicroCUDA reuse each image only while its allocation identity and write epoch remain valid. ZLUDA still prepares each tube launch. A 12-frame RX 9070 XT harness sequence matched the token-disabled layers byte-for-byte; MicroCUDA prepared the six tube images once and skipped the following 66 preparations, and repeated feature recreation remained byte-identical.
 - Tried, no measurable gain: interleaving the WMMAs of independent k32 steps. Skipping windows that lie entirely outside the image was also exact but below the measurement noise (at most 2.4% of the windows of every other tube launch). Not tried: true16 code generation (the half-register conversions would save about 3% of the instructions).

docs/performance.md

@@ -131,3 +131,14 @@ The 16-bit path never re-encoded its operands, so it was already faster than the
 The shim's whole-evaluation GPU time (`D4R_PROFILE`, no per-kernel synchronisation) went from 5.9–6.0 ms to 2.80 ms with the harness's 200 ms pause between frames, and to 2.39–2.43 ms with frames 0–4 ms apart (2000 frames at 4 ms spacing: median 2.39, maximum 2.56). The difference is the driver's clock governor: with the 200 ms pause the GPU is about 9% busy and runs at about 2790 MHz, with frames 0–4 ms apart it is 80–98% busy and runs at 3180–3300 MHz, as it does under a game's rendering load. Other GPU users on the desktop add occasional slower frames to any of these figures; the previous kernels show the same disturbances.
 
 The change is in [native-kernels.md](native-kernels.md#rdna4): the FP8 layers keep their activations as e4m3 bytes, and the downsample kernel is built as wave64 on gfx12. The output image is byte-identical before and after.
+
+DLSS 4.5 (L) on the same card and harness, with enc0 and dec0 as ZLUDA's compile (`sust_only`) and with their native token-lane parts ([native-kernels.md](native-kernels.md#families)). Median GPU time per frame in milliseconds from `D4R_CUDA_KERNEL_PROFILE` (frames 11–30, moving scene, LDR input unless noted); the final frames are byte-identical in every column pair.
+
+| | 1280×720 → 4K | | 1280×720 → 4K, HDR flags | | 1920×1080 → 4K | | accuracy set, 1280×720 → 4K | |
+|---|---|---|---|---|---|---|---|---|
+| Kernel | before | now | before | now | before | now | before | now |
+| enc0 | 1.69 | 0.63 | 1.56 | 0.67 | 2.73 | 1.16 | 1.82 | 0.67 |
+| dec0 | 1.26 | 0.54 | 1.26 | 0.54 | 2.28 | 1.01 | 1.31 | 0.59 |
+| **all kernels** | **5.83** | **4.06** | **5.93** | **4.34** | **8.63** | **5.69** | **5.99** | **4.14** |
+
+L's post kernel (1.2–1.5 ms) is the largest kernel left; it has no matrix multiplies. The 1920×1080 runs were taken in one of the two clock states the driver's governor settles in at that load (about 13% apart), so compare within the pair, not across resolutions.