kernels: build the denoiser's output kernel natively, authorized separately
Ray Reconstruction's eleven DLSS-D Swin layers still run ZLUDA's compile of NVIDIA's PTX; at 1280x720 -> 3840x2160 on gfx1201 they are 30.2 of a 31.6 ms frame. The denoiser's high-fidelity output kernel is the first one with a native replacement: make_ptx.py replaces its whole-texel RGBA16F store (sust.b.2d.v4.b16.zero) through d4r_sust_v4b16, kernels/build.sh builds the pair as the `rr` family, and packaging builds it into every target folder including the RDNA4 FP8 ones. Both libraries define modules named cuda_dldn_engine_*, with different text, so one manifest cannot take the union of their hashes: kernel_manifest.py now takes the denoiser DLL as --rr-dll and lets only its hashes authorize a cuda_dldn_engine_* binary, and package_release.sh passes it when the denoiser is bundled. Verified in the D3D12 harness with Ray Reconstruction at 1280x720 -> 3840x2160: the output kernel drops from 1.44 to 1.00 ms/frame and the 66 MB raw output is byte-identical to the run with every kernel translated. docs/native-kernels.md gains the denoiser family, the per-kernel cost table and the instruction counts that show why the Swin layers are the remaining work.
This commit is contained in:
1 parent
56e1b76622
commit
74ce9d8f9a
7 files changed
+199
-12
No files matched your search
@@ -71,6 +71,26 @@ Other targets (RDNA3, and RDNA4 with `NativeFp8` off) keep the `sust_only` build
|
|||||||
|
|
||||||
Initial validation: a gfx1101 fast L build completed two Ultra Performance harness frames (192×108 → 576×324, DLSS 310.7). Its finite, nonzero RGBA16F output matched the original translated L texture kernels byte for byte with the same native Swin layers. A gfx1201 FP8 accuracy L set compiled successfully; RDNA4 hardware and 4K game output have not been tested. Full packaging rebuilds missing L variants when given an older texture bundle.
|
Initial validation: a gfx1101 fast L build completed two Ultra Performance harness frames (192×108 → 576×324, DLSS 310.7). Its finite, nonzero RGBA16F output matched the original translated L texture kernels byte for byte with the same native Swin layers. A gfx1201 FP8 accuracy L set compiled successfully; RDNA4 hardware and 4K game output have not been tested. Full packaging rebuilds missing L variants when given an older texture bundle.
|
||||||
|
|
||||||
|
**Ray Reconstruction, the denoiser (`kernels/tex` + `cuda_dldn_engine_*`).**
|
||||||
|
- **Network:** `nvngx_dlssd.dll` defines the DLSS-D denoiser: six Swin encoders (`cuda_dldn_engine_swin_enc0_kernel` … `enc5`), five decoders (`dec0` … `dec4`), a high-fidelity kernel-prediction network whose output kernel is `cuda_dldn_engine_hkpn_output_kernel_transformer`, and the luma/auto-exposure/`reduce_sum` helpers. It shares no kernel with the Super Resolution libraries.
|
||||||
|
- **Native today:** the hkpn output kernel (and its `_diamond_wallaby` twin), built by `kernels/build.sh rr` as a wave64 surface-store replacement. `make_ptx.py` replaces `sust.b.2d.v4.b16.zero` — a whole RGBA16F texel — with `d4r_sust_v4b16`; the neural body and the texture reads are unchanged.
|
||||||
|
- **Still translated:** all eleven Swin layers. ZLUDA compiles NVIDIA's PTX at load time, and that compile is what costs the time (see the table below). A native replacement has to recover each layer's attention, norm and merge stages, the way `kernels/m/rrlite_*` does for the upscaler.
|
||||||
|
- **Manifest:** the denoiser DLL is passed to `kernel_manifest.py` as `--rr-dll`, because its hashes may only authorize `cuda_dldn_engine_*` replacements. Without that split, a name both libraries define — `dl4rt_input_kernel` — would let a super-resolution binary be served for the denoiser's different implementation of it.
|
||||||
|
|
||||||
|
Measured on the RX 9070 XT (gfx1201, RDNA4), Ray Reconstruction preset E, 1280×720 → 3840×2160, `D4R_CUDA_KERNEL_PROFILE` in the D3D12 harness, 16 frames after 6 warm-up, with the shipped `[Kernels]` settings (`NativeFp8`, `IgnoreDenormals`; `IgnoreDenormals` alone is worth 11% here and `FastMath` changes nothing):
|
||||||
|
|
||||||
|
| kernel | ms/frame | kernel | ms/frame |
|
||||||
|
|---|---|---|---|
|
||||||
|
| dec0 | 7.71 | dec4 | 1.16 |
|
||||||
|
| enc0 | 6.49 | dec3 | 1.17 |
|
||||||
|
| dec1 | 3.66 | enc4 | 1.02 |
|
||||||
|
| enc1 | 3.48 | enc3 | 0.92 |
|
||||||
|
| dec2 | 2.18 | enc5 | 0.65 |
|
||||||
|
| enc2 | 1.74 | luma, exposure, reduce_sum | 0.05 |
|
||||||
|
| hkpn output (translated / native) | 1.44 / 0.85 | **total** | **31.6** |
|
||||||
|
|
||||||
|
The eleven Swin layers are 30.2 of those 31.6 ms. They are instruction-bound, not MMA-bound: the compiled `enc0` kernel is 83 373 instructions per thread for 536 FP8 WMMAs (0.6% of the stream), and the frame reaches about 6% of the card's FP8 WMMA peak. The stream is 13 k operand-gather shuffles (`ds_bpermute`/`v_perm`/`permlane`), 15 k packed 16-bit ops from the software `cvt.rn.satfinite.e4m3x2.f16x2` requantization, 9 k selects and 13 k ALU-dependency waits. `D4R_ZLUDA_IMPLICIT_MAX_BLOCK` (256 in the shipped config), `D4R_ZLUDA_SCHED_STRATEGY` and a 128-thread bound move the total by under 3%; a 128-thread bound additionally fails the launches of kernels that use larger blocks, and the whole-module wave64 switch produces no launches at all. Removing the gather and the codec — what a native layer does — is the remaining work.
|
||||||
|
|
||||||
## RDNA4
|
## RDNA4
|
||||||
|
|
||||||
`kernels/common/wmma_layout.h` selects the WMMA operand and accumulator layout. gfx12 uses eight f16 values per lane in each K half; the accumulator rows are `i + 8·half`. The gfx11 path retains its original layout. `D4R_WMMA_LAYOUT=12` on gfx11 is a test shim: it exercises gfx12 indexing but executes gfx11 WMMA, so its replay output can be compared byte for byte with the existing gfx11 build.
|
`kernels/common/wmma_layout.h` selects the WMMA operand and accumulator layout. gfx12 uses eight f16 values per lane in each K half; the accumulator rows are `i + 8·half`. The gfx11 path retains its original layout. `D4R_WMMA_LAYOUT=12` on gfx11 is a test shim: it exercises gfx12 indexing but executes gfx11 WMMA, so its replay output can be compared byte for byte with the existing gfx11 build.
|
||||||
|
|||||||
+21
-2
@@ -2,7 +2,7 @@
|
|||||||
# Builds the native replacements for DLSS kernels into one directory that ZLUDA serves them from
|
# Builds the native replacements for DLSS kernels into one directory that ZLUDA serves them from
|
||||||
# (D4R_ZLUDA_NATIVE_DIR, or NativeKernelDirFast in d4r.ini). See docs/native-kernels.md.
|
# (D4R_ZLUDA_NATIVE_DIR, or NativeKernelDirFast in d4r.ini). See docs/native-kernels.md.
|
||||||
#
|
#
|
||||||
# usage: kernels/build.sh [all|k|l|m|tex] [OUT_DIR] (default: all, kernels/out/native)
|
# usage: kernels/build.sh [all|k|l|m|tex|rr] [OUT_DIR] (default: all, kernels/out/native)
|
||||||
#
|
#
|
||||||
# D4R_ROCM_DIR ROCm installation with clang and the HIP device libraries (default /opt/rocm)
|
# D4R_ROCM_DIR ROCm installation with clang and the HIP device libraries (default /opt/rocm)
|
||||||
# D4R_GPU_ARCH target GPU (default gfx1101): an RDNA3 (gfx110x) or RDNA4 (gfx120x) target; the kernels
|
# D4R_GPU_ARCH target GPU (default gfx1101): an RDNA3 (gfx110x) or RDNA4 (gfx120x) target; the kernels
|
||||||
@@ -17,6 +17,8 @@
|
|||||||
# D4R_ZLUDA_EMIT ZLUDA's d4r_emit (patches/zluda applied; `cargo build --release -p ptx --example d4r_emit`),
|
# D4R_ZLUDA_EMIT ZLUDA's d4r_emit (patches/zluda applied; `cargo build --release -p ptx --example d4r_emit`),
|
||||||
# which compiles PTX offline for D4R_GPU_ARCH (no GPU of that kind needed)
|
# which compiles PTX offline for D4R_GPU_ARCH (no GPU of that kind needed)
|
||||||
# D4R_DLSS_PTX_DIR optional pre-extracted PTX directory for parallel per-target builds
|
# D4R_DLSS_PTX_DIR optional pre-extracted PTX directory for parallel per-target builds
|
||||||
|
# Ray Reconstruction (rr) additionally needs D4R_DLSSD_DLL (CUDA-capable
|
||||||
|
# nvngx_dlssd.dll); D4R_DLSSD_PTX_DIR may provide its pre-extracted modules.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
shopt -s extglob
|
shopt -s extglob
|
||||||
|
|
||||||
@@ -38,7 +40,7 @@ esac
|
|||||||
CLANG="$ROCM/lib/llvm/bin/clang++"
|
CLANG="$ROCM/lib/llvm/bin/clang++"
|
||||||
[[ -x "$CLANG" ]] || { echo "clang++ not found in $ROCM/lib/llvm/bin (set D4R_ROCM_DIR)" >&2; exit 2; }
|
[[ -x "$CLANG" ]] || { echo "clang++ not found in $ROCM/lib/llvm/bin (set D4R_ROCM_DIR)" >&2; exit 2; }
|
||||||
mkdir -p "$OUT"
|
mkdir -p "$OUT"
|
||||||
case "$WHAT" in all|k|l|m|tex) ;; *) echo "unknown kernel family: $WHAT" >&2; exit 2 ;; esac
|
case "$WHAT" in all|k|l|m|tex|rr) ;; *) echo "unknown kernel family: $WHAT" >&2; exit 2 ;; esac
|
||||||
# Never certify a directory containing older fast binaries as an accuracy set.
|
# Never certify a directory containing older fast binaries as an accuracy set.
|
||||||
if [[ "$ACCURACY" == 1 ]] && compgen -G "$OUT/*.hsaco" >/dev/null &&
|
if [[ "$ACCURACY" == 1 ]] && compgen -G "$OUT/*.hsaco" >/dev/null &&
|
||||||
{ [[ ! -f "$OUT/d4r-accuracy.txt" ]] || [[ "$(cat "$OUT/d4r-accuracy.txt")" != 1 ]]; }; then
|
{ [[ ! -f "$OUT/d4r-accuracy.txt" ]] || [[ "$(cat "$OUT/d4r-accuracy.txt")" != 1 ]]; }; then
|
||||||
@@ -185,5 +187,22 @@ if [[ "$WHAT" == all || "$WHAT" == tex || "$WHAT" == l ]]; then
|
|||||||
done
|
done
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
if [[ "$WHAT" == rr || ( "$WHAT" == all && -n "${D4R_DLSSD_DLL:-}" ) ]]; then
|
||||||
|
echo "== Ray Reconstruction texture kernels"
|
||||||
|
: "${D4R_DLSSD_DLL:?set D4R_DLSSD_DLL to the CUDA-capable nvngx_dlssd.dll}"
|
||||||
|
: "${D4R_ZLUDA_EMIT:?set D4R_ZLUDA_EMIT to d4r_emit from a patched ZLUDA build}"
|
||||||
|
RR_PTX_DIR="${D4R_DLSSD_PTX_DIR:-$HERE/extracted/rr}"
|
||||||
|
if [[ -z "${D4R_DLSSD_PTX_DIR:-}" ]]; then
|
||||||
|
python3 "$HERE/tools/extract_dlss_ptx.py" "$D4R_DLSSD_DLL" "$RR_PTX_DIR"
|
||||||
|
elif [[ ! -d "$RR_PTX_DIR" ]]; then
|
||||||
|
echo "missing extracted RR PTX directory: $RR_PTX_DIR" >&2; exit 2
|
||||||
|
fi
|
||||||
|
for kernel in cuda_dldn_engine_hkpn_output_kernel_transformer \
|
||||||
|
cuda_dldn_engine_hkpn_output_kernel_transformer_diamond_wallaby; do
|
||||||
|
D4R_PREFER_ACCURACY="$ACCURACY" D4R_ROCM_DIR="$ROCM" D4R_GPU_ARCH="$ARCH" \
|
||||||
|
D4R_ZLUDA_WAVE64="$((1 - ACCURACY))" D4R_TEX_FP8="$FP8" \
|
||||||
|
D4R_DLSS_PTX_DIR="$RR_PTX_DIR" "$HERE/tex/build_tex.sh" "$kernel" sust_only "$OUT"
|
||||||
|
done
|
||||||
|
fi
|
||||||
[[ "$ACCURACY" == 1 ]] && printf '1\n' > "$OUT/d4r-accuracy.txt"
|
[[ "$ACCURACY" == 1 ]] && printf '1\n' > "$OUT/d4r-accuracy.txt"
|
||||||
echo "native kernels in $OUT"
|
echo "native kernels in $OUT"
|
||||||
+28
-3
@@ -99,6 +99,13 @@ SUST_EXTERN = """.extern .func d4r_sust_v2b16
|
|||||||
)
|
)
|
||||||
;
|
;
|
||||||
"""
|
"""
|
||||||
|
SUST4_EXTERN = """.extern .func d4r_sust_v4b16
|
||||||
|
(
|
||||||
|
.param .b64 d4r_h0, .param .b32 d4r_h1, .param .b32 d4r_h2, .param .b16 d4r_h3,
|
||||||
|
.param .b16 d4r_h4, .param .b16 d4r_h5, .param .b16 d4r_h6
|
||||||
|
)
|
||||||
|
;
|
||||||
|
"""
|
||||||
SUSTP_EXTERN = """.extern .func d4r_sust_p_v4b32
|
SUSTP_EXTERN = """.extern .func d4r_sust_p_v4b32
|
||||||
(
|
(
|
||||||
.param .b64 d4r_t0, .param .b32 d4r_t1, .param .b32 d4r_t2, .param .b32 d4r_t3, .param .b32 d4r_t4,
|
.param .b64 d4r_t0, .param .b32 d4r_t1, .param .b32 d4r_t2, .param .b32 d4r_t3, .param .b32 d4r_t4,
|
||||||
@@ -115,14 +122,31 @@ SUSTB32_EXTERN = """.extern .func d4r_sust_b32
|
|||||||
"""
|
"""
|
||||||
SUSTB32_RE = re.compile(r"^sust\.b\.2d\.b32\.zero \[(%rd\d+), \{(%r\d+),(%r\d+)\}\], \{(%r\d+)\};$")
|
SUSTB32_RE = re.compile(r"^sust\.b\.2d\.b32\.zero \[(%rd\d+), \{(%r\d+),(%r\d+)\}\], \{(%r\d+)\};$")
|
||||||
SUST_RE = re.compile(r"^sust\.b\.2d\.v2\.b16\.zero \[(%rd\d+), \{(%r\d+),(%r\d+)\}\], \{(%rs\d+),(%rs\d+)\};$")
|
SUST_RE = re.compile(r"^sust\.b\.2d\.v2\.b16\.zero \[(%rd\d+), \{(%r\d+),(%r\d+)\}\], \{(%rs\d+),(%rs\d+)\};$")
|
||||||
|
SUST4_RE = re.compile(r"^sust\.b\.2d\.v4\.b16\.zero \[(%rd\d+), \{(%r\d+),\s*(%r\d+)\}\], \{(%rs\d+),\s*(%rs\d+),\s*(%rs\d+),\s*(%rs\d+)\};$")
|
||||||
|
|
||||||
|
|
||||||
SUST_KINDS = set(os.environ.get("D4R_SUST_KINDS", "b16,p,b32").split(","))
|
SUST_KINDS = set(os.environ.get("D4R_SUST_KINDS", "b16,p,b32").split(","))
|
||||||
|
|
||||||
|
|
||||||
def replace_sust(lines):
|
def replace_sust(lines):
|
||||||
out, n = [], {"b16": 0, "p": 0, "b32": 0}
|
out, n = [], {"b16": 0, "v4b16": 0, "p": 0, "b32": 0}
|
||||||
for line in lines:
|
for line in lines:
|
||||||
|
mh = SUST4_RE.match(line.strip()) if "b16" in SUST_KINDS else None
|
||||||
|
if mh:
|
||||||
|
s, x, y, *v = mh.groups()
|
||||||
|
st = "\n".join(f"\t.param .b16 v{i};\n\tst.param.b16 [v{i}+0], {r};" for i, r in enumerate(v))
|
||||||
|
out.append(f"""{{
|
||||||
|
.param .b64 u0;
|
||||||
|
st.param.b64 [u0+0], {s};
|
||||||
|
.param .b32 u1;
|
||||||
|
st.param.b32 [u1+0], {x};
|
||||||
|
.param .b32 u2;
|
||||||
|
st.param.b32 [u2+0], {y};
|
||||||
|
{st}
|
||||||
|
call d4r_sust_v4b16, (u0, u1, u2, v0, v1, v2, v3);
|
||||||
|
}}""")
|
||||||
|
n["v4b16"] += 1
|
||||||
|
continue
|
||||||
mb = SUSTB32_RE.match(line.strip()) if "b32" in SUST_KINDS else None
|
mb = SUSTB32_RE.match(line.strip()) if "b32" in SUST_KINDS else None
|
||||||
if mb:
|
if mb:
|
||||||
s, x, y, v = mb.groups()
|
s, x, y, v = mb.groups()
|
||||||
@@ -414,7 +438,8 @@ ENC0_REF = "rrlite_enc0_4x4_mvhi_hdr_folded"
|
|||||||
SUST_ONLY_RE = re.compile(r"^(hiluma_engine_output_depth(inv|reg)_mv(hi|lo)_(hdr|ldr)(_max)?_v[12]_rel|"
|
SUST_ONLY_RE = re.compile(r"^(hiluma_engine_output_depth(inv|reg)_mv(hi|lo)_(hdr|ldr)(_max)?_v[12]_rel|"
|
||||||
r"rrlite_post_3_[12]_mv(hi|lo)_(hdr|ldr)(_folded)?|"
|
r"rrlite_post_3_[12]_mv(hi|lo)_(hdr|ldr)(_folded)?|"
|
||||||
r"rrlite_enc0_4x4_mv(hi|lo)_(hdr|ldr)|rrlite_dec0_4x4|"
|
r"rrlite_enc0_4x4_mv(hi|lo)_(hdr|ldr)|rrlite_dec0_4x4|"
|
||||||
r"rrlite_downsample_kernel_(static|dynamic)_(hdr|ldr))$")
|
r"rrlite_downsample_kernel_(static|dynamic)_(hdr|ldr)|"
|
||||||
|
r"cuda_dldn_engine_hkpn_output_kernel_transformer(_diamond_wallaby)?)$")
|
||||||
ENC0_RE = re.compile(r"^rrlite_enc0_4x4_mv(hi|lo)_(hdr|ldr)_folded$")
|
ENC0_RE = re.compile(r"^rrlite_enc0_4x4_mv(hi|lo)_(hdr|ldr)_folded$")
|
||||||
|
|
||||||
|
|
||||||
@@ -536,7 +561,7 @@ def main():
|
|||||||
if n["dec8"] != k["dec8"]:
|
if n["dec8"] != k["dec8"]:
|
||||||
sys.exit(f"{name}: {n['dec8']} e4m3 decodes, expected {k['dec8']}")
|
sys.exit(f"{name}: {n['dec8']} e4m3 decodes, expected {k['dec8']}")
|
||||||
k = dict(k, extern=k["extern"] + DEC8_EXTERN)
|
k = dict(k, extern=k["extern"] + DEC8_EXTERN)
|
||||||
extern = k["extern"] + (SUST_EXTERN if n["b16"] else "") + (SUSTP_EXTERN if n["p"] else "") + (SUSTB32_EXTERN if n["b32"] else "")
|
extern = k["extern"] + (SUST_EXTERN if n["b16"] else "") + (SUST4_EXTERN if n.get("v4b16") else "") + (SUSTP_EXTERN if n["p"] else "") + (SUSTB32_EXTERN if n["b32"] else "")
|
||||||
text = "\n".join(head[:entry]) + "\n" + extern + "\n".join(body + tail)
|
text = "\n".join(head[:entry]) + "\n" + extern + "\n".join(body + tail)
|
||||||
out.write_text(text)
|
out.write_text(text)
|
||||||
print(f"{out}: {len(text.splitlines())} lines, surface stores replaced: {n}")
|
print(f"{out}: {len(text.splitlines())} lines, surface stores replaced: {n}")
|
||||||
|
|||||||
@@ -261,6 +261,39 @@ DEV __attribute__((always_inline)) void d4r_sust_v2b16(uint64_t surface, int32_t
|
|||||||
sust_v2b16_generic(surface, x, y, a, b);
|
sust_v2b16_generic(surface, x, y, a, b);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
typedef int16_t v4s __attribute__((ext_vector_type(4)));
|
||||||
|
extern "C" __attribute__((device)) void __zluda_ptx_impl_surfobj_b_2d_v4_b16_zero(uint64_t surface, v2i coord, v4s value);
|
||||||
|
|
||||||
|
__attribute__((device, noinline)) static void sust_v4b16_generic(uint64_t surface, int32_t x, int32_t y,
|
||||||
|
uint16_t a, uint16_t b, uint16_t c, uint16_t d)
|
||||||
|
{
|
||||||
|
__zluda_ptx_impl_surfobj_b_2d_v4_b16_zero(surface, (v2i){x, y},
|
||||||
|
(v4s){(int16_t)a, (int16_t)b, (int16_t)c, (int16_t)d});
|
||||||
|
}
|
||||||
|
|
||||||
|
FP_BODY void sust_rgba16f_body(tsharp_t* image, int32_t px, int32_t y,
|
||||||
|
uint16_t a, uint16_t b, uint16_t c, uint16_t d)
|
||||||
|
{
|
||||||
|
__ockl_image_store_2D(image, (v2i){px, y},
|
||||||
|
(v4f){half_bits_to_float(a), half_bits_to_float(b),
|
||||||
|
half_bits_to_float(c), half_bits_to_float(d)});
|
||||||
|
}
|
||||||
|
|
||||||
|
// RR stores complete RGBA16F texels. No channel reads or partial-pixel merges
|
||||||
|
// are needed; other layouts retain the generic byte-addressed store.
|
||||||
|
DEV __attribute__((always_inline)) void d4r_sust_v4b16(uint64_t surface, int32_t x, int32_t y,
|
||||||
|
uint16_t a, uint16_t b, uint16_t c, uint16_t d)
|
||||||
|
{
|
||||||
|
tsharp_t* image = (tsharp_t*)surface;
|
||||||
|
if (__ockl_image_channel_data_type_2D(image) == 14 && __ockl_image_channel_order_2D(image) == 8 && (x & 7) == 0)
|
||||||
|
{
|
||||||
|
if (x >= 0)
|
||||||
|
sust_rgba16f_body(image, x >> 3, y, a, b, c, d);
|
||||||
|
}
|
||||||
|
else
|
||||||
|
sust_v4b16_generic(surface, x, y, a, b, c, d);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
// f32 comparisons on the bits (no fcmp: kernels ZLUDA compiles in strict-FP mode cannot lower
|
// f32 comparisons on the bits (no fcmp: kernels ZLUDA compiles in strict-FP mode cannot lower
|
||||||
// constrained fcmp); same results as the float compares, -0.0 included
|
// constrained fcmp); same results as the float compares, -0.0 included
|
||||||
|
|||||||
@@ -1,14 +1,19 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Write the d4r-kernels.txt manifest of a native kernel directory.
|
"""Write the d4r-kernels.txt manifest of a native kernel directory.
|
||||||
|
|
||||||
usage: kernel_manifest.py KERNEL_DIR NVNGX_DLSS_DLL [NVNGX_DLSS_DLL...]
|
usage: kernel_manifest.py KERNEL_DIR [NVNGX_DLSS_DLL...] [--rr-dll NVNGX_DLSSD_DLL]
|
||||||
|
|
||||||
For every NAME.hsaco in KERNEL_DIR, finds the PTX module of each DLL that defines `.entry NAME` and
|
For every NAME.hsaco in KERNEL_DIR, finds the PTX module of each DLL that defines `.entry NAME` and
|
||||||
lists the FNV-1a 64 hash of the module's text (trailing NUL bytes removed), one "NAME HASH" line per
|
lists the FNV-1a 64 hash of the module's text (trailing NUL bytes removed), one "NAME HASH" line per
|
||||||
distinct hash. The Wine CUDA bridge serves a native kernel only while DLSS loads a module with a
|
distinct hash. The Wine CUDA bridge serves a native kernel only while DLSS loads a module with a
|
||||||
listed hash, so a DLSS version that changed the kernel runs ZLUDA's own compile of it instead.
|
listed hash, so a DLSS version that changed the kernel runs ZLUDA's own compile of it instead.
|
||||||
The manifest is written to KERNEL_DIR/d4r-kernels.txt; it holds hashes, not NVIDIA code.
|
The manifest is written to KERNEL_DIR/d4r-kernels.txt; it holds hashes, not NVIDIA code.
|
||||||
|
RR DLLs are explicit: only a DLL passed with --rr-dll may authorize a cuda_dldn_engine_* replacement,
|
||||||
|
and the Super Resolution DLLs passed positionally authorize everything else. Both libraries define
|
||||||
|
cuda_dldn_engine_* modules of their own (the shared capture and exposure helpers), so without the split
|
||||||
|
a Super Resolution hash could stand in for the denoiser's different implementation of the same name.
|
||||||
"""
|
"""
|
||||||
|
import argparse
|
||||||
import hashlib
|
import hashlib
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
@@ -41,18 +46,24 @@ def module_hashes(dll: str) -> dict:
|
|||||||
|
|
||||||
|
|
||||||
def main(argv):
|
def main(argv):
|
||||||
if len(argv) < 3:
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
sys.exit(__doc__)
|
parser.add_argument("directory")
|
||||||
directory = argv[1]
|
parser.add_argument("dlls", nargs="*")
|
||||||
|
parser.add_argument("--rr-dll", action="append", default=[])
|
||||||
|
args = parser.parse_args(argv[1:])
|
||||||
|
if not args.dlls and not args.rr_dll:
|
||||||
|
parser.error("at least one SR DLL or --rr-dll is required")
|
||||||
|
directory = args.directory
|
||||||
names = sorted(name[:-len('.hsaco')] for name in os.listdir(directory) if name.endswith('.hsaco'))
|
names = sorted(name[:-len('.hsaco')] for name in os.listdir(directory) if name.endswith('.hsaco'))
|
||||||
lines = ['# d4r native kernels: NAME and the FNV-1a 64 hash of the DLSS PTX module each was written for',
|
lines = ['# d4r native kernels: NAME and the FNV-1a 64 hash of the DLSS PTX module each was written for',
|
||||||
'# (written by kernels/tools/kernel_manifest.py from:']
|
'# (written by kernels/tools/kernel_manifest.py from:']
|
||||||
listed = {name: set() for name in names}
|
listed = {name: set() for name in names}
|
||||||
for dll in argv[2:]:
|
for dll, reconstruction in [(dll, False) for dll in args.dlls] + [(dll, True) for dll in args.rr_dll]:
|
||||||
digest = hashlib.sha256(open(dll, 'rb').read()).hexdigest()[:16]
|
digest = hashlib.sha256(open(dll, 'rb').read()).hexdigest()[:16]
|
||||||
lines.append(f'# {os.path.basename(dll)} sha256 {digest}...)')
|
lines.append(f'# {os.path.basename(dll)} sha256 {digest}...)')
|
||||||
hashes = module_hashes(dll)
|
hashes = module_hashes(dll)
|
||||||
for name in names:
|
for name in names:
|
||||||
|
if name.startswith("cuda_dldn_engine_") == reconstruction:
|
||||||
listed[name] |= hashes.get(name, set())
|
listed[name] |= hashes.get(name, set())
|
||||||
missing = [name for name in names if not listed[name]]
|
missing = [name for name in names if not listed[name]]
|
||||||
if missing:
|
if missing:
|
||||||
|
|||||||
@@ -137,6 +137,8 @@ if [[ -n "${D4R_BUNDLE_ENGINE:-}" ]]; then
|
|||||||
printf 'The native engine needs weights.bin, offsets.bin and lut.bin here (see D4R_README.txt).\r\n' > "$STAGE/d4r/engine/k/README.txt"
|
printf 'The native engine needs weights.bin, offsets.bin and lut.bin here (see D4R_README.txt).\r\n' > "$STAGE/d4r/engine/k/README.txt"
|
||||||
fi
|
fi
|
||||||
IFS=: read -r -a DLLS <<< "$D4R_DLSS_DLLS"
|
IFS=: read -r -a DLLS <<< "$D4R_DLSS_DLLS"
|
||||||
|
RR_DLLS=()
|
||||||
|
[[ -z "${D4R_BUNDLE_DLSSD:-}" ]] || RR_DLLS=(--rr-dll "$D4R_BUNDLE_DLSSD")
|
||||||
# one folder per target; RDNA4 targets also get <target>-fp8 (native FP8 WMMA, d4r.ini NativeFp8), which the
|
# one folder per target; RDNA4 targets also get <target>-fp8 (native FP8 WMMA, d4r.ini NativeFp8), which the
|
||||||
# bridge serves instead when that setting is on
|
# bridge serves instead when that setting is on
|
||||||
folders=()
|
folders=()
|
||||||
@@ -162,6 +164,19 @@ ensure_l_textures() {
|
|||||||
D4R_DLSS_DLL="$D4R_BUNDLE_DLSS" "$ROOT/kernels/build.sh" l "$dir" >/dev/null
|
D4R_DLSS_DLL="$D4R_BUNDLE_DLSS" "$ROOT/kernels/build.sh" l "$dir" >/dev/null
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
ensure_rr_textures() {
|
||||||
|
local dir="$1" accuracy="$2" missing=0 name
|
||||||
|
[[ -n "${D4R_BUNDLE_DLSSD:-}" ]] || return 0
|
||||||
|
for name in cuda_dldn_engine_hkpn_output_kernel_transformer \
|
||||||
|
cuda_dldn_engine_hkpn_output_kernel_transformer_diamond_wallaby; do
|
||||||
|
[[ -f "$dir/$name.hsaco" ]] || missing=1
|
||||||
|
done
|
||||||
|
if [[ "$missing" == 1 ]]; then
|
||||||
|
: "${D4R_ZLUDA_EMIT:?set D4R_ZLUDA_EMIT or supply the RR texture variants in D4R_BUNDLE_TEX}"
|
||||||
|
D4R_PREFER_ACCURACY="$accuracy" D4R_ROCM_DIR="$ROCM" D4R_GPU_ARCH="$arch" D4R_NATIVE_FP8="$fp8" \
|
||||||
|
D4R_DLSSD_DLL="$D4R_BUNDLE_DLSSD" "$ROOT/kernels/build.sh" rr "$dir" >/dev/null
|
||||||
|
fi
|
||||||
|
}
|
||||||
build_target() {
|
build_target() {
|
||||||
local folder="$1" arch fp8 tex_dir accurate accurate_tex f
|
local folder="$1" arch fp8 tex_dir accurate accurate_tex f
|
||||||
arch="${folder%-fp8}"
|
arch="${folder%-fp8}"
|
||||||
@@ -185,7 +200,8 @@ build_target() {
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
[[ "$VARIANT" == clean ]] || ensure_l_textures "$STAGE/d4r/kernels/$folder" 0
|
[[ "$VARIANT" == clean ]] || ensure_l_textures "$STAGE/d4r/kernels/$folder" 0
|
||||||
python3 "$ROOT/kernels/tools/kernel_manifest.py" "$STAGE/d4r/kernels/$folder" "${DLLS[@]}"
|
[[ "$VARIANT" == clean ]] || ensure_rr_textures "$STAGE/d4r/kernels/$folder" 0
|
||||||
|
python3 "$ROOT/kernels/tools/kernel_manifest.py" "$STAGE/d4r/kernels/$folder" "${DLLS[@]}" "${RR_DLLS[@]}"
|
||||||
|
|
||||||
# Every release target also gets conservative network and texture variants. Do not copy fast
|
# Every release target also gets conservative network and texture variants. Do not copy fast
|
||||||
# texture objects into this set: their compiler policy is fixed inside the binary.
|
# texture objects into this set: their compiler policy is fixed inside the binary.
|
||||||
@@ -207,6 +223,7 @@ build_target() {
|
|||||||
fi
|
fi
|
||||||
if [[ "$VARIANT" != clean ]]; then
|
if [[ "$VARIANT" != clean ]]; then
|
||||||
ensure_l_textures "$accurate" 1
|
ensure_l_textures "$accurate" 1
|
||||||
|
ensure_rr_textures "$accurate" 1
|
||||||
for f in "$STAGE/d4r/kernels/$folder"/*.hsaco; do
|
for f in "$STAGE/d4r/kernels/$folder"/*.hsaco; do
|
||||||
# the fast set's native hiluma input kernels have no accuracy build: that set keeps ZLUDA's compile
|
# the fast set's native hiluma input kernels have no accuracy build: that set keeps ZLUDA's compile
|
||||||
[[ "$(basename "$f")" == hiluma_engine_input_* ]] && continue
|
[[ "$(basename "$f")" == hiluma_engine_input_* ]] && continue
|
||||||
@@ -216,7 +233,7 @@ build_target() {
|
|||||||
done
|
done
|
||||||
fi
|
fi
|
||||||
rm -f "$accurate"/*.resolution.txt
|
rm -f "$accurate"/*.resolution.txt
|
||||||
python3 "$ROOT/kernels/tools/kernel_manifest.py" "$accurate" "${DLLS[@]}"
|
python3 "$ROOT/kernels/tools/kernel_manifest.py" "$accurate" "${DLLS[@]}" "${RR_DLLS[@]}"
|
||||||
}
|
}
|
||||||
kernel_jobs="${D4R_PACKAGE_KERNEL_JOBS:-1}"
|
kernel_jobs="${D4R_PACKAGE_KERNEL_JOBS:-1}"
|
||||||
[[ "$kernel_jobs" =~ ^[1-9][0-9]*$ ]] || { echo "D4R_PACKAGE_KERNEL_JOBS must be positive" >&2; exit 2; }
|
[[ "$kernel_jobs" =~ ^[1-9][0-9]*$ ]] || { echo "D4R_PACKAGE_KERNEL_JOBS must be positive" >&2; exit 2; }
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
"""Native SR code must never be authorized by a colliding denoiser entry."""
|
||||||
|
import importlib.util
|
||||||
|
from pathlib import Path
|
||||||
|
import struct
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
TOOLS = ROOT / 'kernels/tools'
|
||||||
|
sys.path.insert(0, str(TOOLS))
|
||||||
|
spec = importlib.util.spec_from_file_location('d4r_kernel_manifest', TOOLS / 'kernel_manifest.py')
|
||||||
|
manifest = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(manifest)
|
||||||
|
|
||||||
|
|
||||||
|
def fatbin(ptx):
|
||||||
|
entry = bytearray(64)
|
||||||
|
struct.pack_into('<HHIQ', entry, 0, 1, 0, 64, len(ptx))
|
||||||
|
payload = entry + ptx
|
||||||
|
return struct.pack('<IHHQ', 0xBA55ED50, 1, 16, len(payload)) + payload
|
||||||
|
|
||||||
|
|
||||||
|
class KernelManifestTests(unittest.TestCase):
|
||||||
|
def test_denoiser_owned_name_comes_only_from_the_denoiser_dll(self):
|
||||||
|
# A cuda_dldn_engine_* kernel may only be authorized by the DLL passed as --rr-dll; the
|
||||||
|
# Super Resolution libraries define names of that shape too, and their modules differ.
|
||||||
|
sr = b'.visible .entry dl4rt_input_kernel() { ret; }\n'
|
||||||
|
rr = (b'.visible .entry dl4rt_input_kernel() { mov.u32 %r1, 1; ret; }\n'
|
||||||
|
b'.visible .entry cuda_dldn_engine_swin_enc1_kernel() { ret; }\n')
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
root = Path(directory)
|
||||||
|
(root / 'sr.dll').write_bytes(fatbin(sr))
|
||||||
|
(root / 'rr.dll').write_bytes(fatbin(rr))
|
||||||
|
for name in ('dl4rt_input_kernel', 'cuda_dldn_engine_swin_enc1_kernel'):
|
||||||
|
(root / (name + '.hsaco')).touch()
|
||||||
|
subprocess.run([sys.executable, str(TOOLS / 'kernel_manifest.py'), str(root),
|
||||||
|
str(root / 'sr.dll'), '--rr-dll', str(root / 'rr.dll')],
|
||||||
|
check=True, capture_output=True)
|
||||||
|
rows = [line.split() for line in (root / 'd4r-kernels.txt').read_text().splitlines()
|
||||||
|
if line and not line.startswith('#')]
|
||||||
|
self.assertEqual(rows, [
|
||||||
|
['cuda_dldn_engine_swin_enc1_kernel', f'{manifest.fnv1a64(rr):016x}'],
|
||||||
|
['dl4rt_input_kernel', f'{manifest.fnv1a64(sr):016x}'],
|
||||||
|
])
|
||||||
|
|
||||||
|
def test_rr_only_input_refuses_unproven_sr_binary(self):
|
||||||
|
rr = b'.visible .entry dl4rt_input_kernel() { ret; }\n'
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
root = Path(directory)
|
||||||
|
(root / 'rr.dll').write_bytes(fatbin(rr))
|
||||||
|
(root / 'dl4rt_input_kernel.hsaco').touch()
|
||||||
|
result = subprocess.run([sys.executable, str(TOOLS / 'kernel_manifest.py'), str(root),
|
||||||
|
'--rr-dll', str(root / 'rr.dll')], capture_output=True, text=True)
|
||||||
|
self.assertNotEqual(result.returncode, 0)
|
||||||
|
self.assertIn('no DLL defines dl4rt_input_kernel', result.stderr)
|
||||||
|
self.assertFalse((root / 'd4r-kernels.txt').exists())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
||||||
Reference in new issue
Block a user