kernels/tex: native token-lane parts for preset L's enc0 and dec0

L's unfolded enc0 and dec0 are still ZLUDA's compile of NVIDIA's PTX and cost
2.95 ms of the preset's 5.86 ms per frame (RX 9070 XT, 1280x720 -> 4K). The
embedding, the patch expand, the Swin block (32 channels, 2 heads, at pixel
resolution) and the patch merge are now native token-lane code; the texture
work around them stays ZLUDA's compile.

enc0l_tail.hip cuts in at the barrier after the feature staging and dec0l_block.hip
before the output code that reads the block's f16 result out of the last MMAs; make_ptx.py
splices either in when D4R_TEX_SRC names it and build.sh selects both for the gfx12
native-FP8 builds. swin_tex.h carries the indexing the two tails share; swin_tok.h gains
the pixel-order token layout (SWIN_TOK_NATURAL), ZLUDA's attention key order
(SWIN_PV_ZLUDA_ORDER) and swin_tok_core, which the encoded tests reuse. swin_common.h
takes the block weight layout (moved out of swin_block.h) and the exact and approximate
reciprocal / mix forms the tails need without the HIP headers (D4R_NO_HIP_RUNTIME).

The f32 sums stay in ZLUDA's order (K slots of the FP8 WMMAs, key order of the f16
attention product) and rsqrt/rcp use the hardware approximations ZLUDA lowers to, so the
harness's final frame is byte-identical to the translated kernels: all four enc0 flag
variants, 720p and 1080p render, 1440p and 4K output, static and moving scenes, 100
frames, and the accuracy set. With M's own sum order about one token in 20,000 differed.
M's layers compile to the same instructions and its frame is byte-identical.

enc0 1.69 -> 0.63 ms and dec0 1.26 -> 0.54 ms at 720p->4K, 2.73 -> 1.16 and 2.28 -> 1.01
at 1080p->4K; the preset 5.83 -> 4.06 and 8.63 -> 5.69 ms. -DD4R_SWIN_K32_MIX rounds the
f16 products earlier for 0.56 and 0.49 ms and is not byte-identical, so it is not built by
default. The repo's tests pass (36, 3 skipped).
This commit is contained in:
woutermans committed 2026-10-08 01:08:34 +02:00
1 parent 0a12b6ce16
commit 8813ea723c
11 files changed
+885 -151

No files matched your search

+8 -2
View File
@@ -112,10 +112,16 @@ if [[ "$WHAT" == all || "$WHAT" == tex || "$WHAT" == l ]]; then
fi
# L's unfolded variants keep their complete neural arithmetic. The folded M
# GEMM heads/tails are incompatible; substitute only the surface stores.
specs+=(rrlite_dec0_4x4:sust_only)
# RDNA4 native FP8: enc0 after its feature staging (embedding, Swin block, patch merge: enc0l_tail.hip) and
# dec0 before its output code (patch expand, Swin block: dec0l_block.hip) are native token-lane code in the
# order of ZLUDA's lowering. Harness frames are byte-identical to the sust_only builds (RX 9070 XT,
# 1280x720 render: enc0 1.71 -> 0.63 ms, dec0 1.26 -> 0.54 ms).
enc0l=sust_only dec0l=sust_only
[[ "$ARCH" == gfx12* && "$FP8" == 1 ]] && enc0l=enc0l_tail dec0l=dec0l_block
specs+=("rrlite_dec0_4x4:$dec0l")
for mv in mvhi mvlo; do
for range in hdr ldr; do
specs+=("rrlite_enc0_4x4_${mv}_${range}:sust_only")
specs+=("rrlite_enc0_4x4_${mv}_${range}:$enc0l")
for v in 3_1 3_2; do specs+=("rrlite_post_${v}_${mv}_${range}:sust_only:w64"); done
done
done