Per-layer staged-image engine for the eleven translated denoiser Swin
layers: descriptors (verified offsets per phase), cut recipes that splice an
isolated native phase into NVIDIA's PTX, wave templates, epilogue notes and
GPU probes. The isolated phase-8 splice produces a byte-identical final
4K frame but is slower than the translated body (14.90 vs 6.59 ms) - a
correctness experiment, not a shipping improvement; docs/rr-engine.md
records the design and the next direct-converter win (B operands built
from transposed accumulators).
Patch 0011 merges a complete accumulator-register pair overwrite into one
shadow-layout conversion (reads, rewritten shadow accesses and block
boundaries prevent it; partial writes keep read/modify/write), and requests
eight waves per SIMD for gfx12 native-FP8 functions. Patch 0012 keeps signs
in the native conversion: finite overflow clamped, NaNs mapped to signed
infinity, post-conversion sign masks gone; plain and ReLU remain
exhaustive-table identical to the software codec.
Eighty-frame harness at 720p->4K, excluding the first twenty: control
24.764 ms, coalescing 23.340 ms, plus eight waves 20.135 ms, plus signed
conversion 16.234 ms (15.144 ms with the harness idle gap removed); all
eighty saved frames byte-identical to the matching control. Twelve- and
ten-wave budgets regress (25.56/24.61 ms); fast math changes the image
(18.92 ms) and is not selected.
Patch 0009 lowers half-to-e4m3 (plain and ReLU) through hardware packed FP8
conversion on gfx12 with explicit saturation and sign restoration; patch 0010
keeps native FP8 operands packed through the NVIDIA-to-WMMA lane gathers.
Same-runtime software control 33.90 ms vs native 26.28/26.65 ms; packed
gathers bring the twenty-frame run to 25.33 ms, all frames byte-identical.
The eleven cuda_dldn_engine_swin_* layers have to be written natively before
the denoiser can come close to 5 ms, and that needs their layout: which maximal
runs of mma exist, what tile each covers, and which bytes of the 54 KB weight
image it reads.
rr_layer_spec.py derives that from the extracted PTX and nothing else - no GPU,
no dependencies beyond the standard library. A phase is one maximal run of
adjacent mma statements; M, N and K come from the number of distinct A and B
fragments and the length of the accumulator chains partitioned by their D-to-C
dataflow, rather than from a guess at the geometry; and each phase reports the
weight and bias byte offsets it reads and the size of the epilogue that follows.
Statement ranges carry their physical PTX line, so every claim can be checked
with sed.
enc0 comes out as 22 phases: two stem projections (M96 N32 K32), the fused Q/V
projection (M96 N32 K32 x2), the score phase (M64 N96 K32, bias 8384..20172 -
a 24-tile 16x16-per-head table), the banded P*V phase (M192 N96 K96, no weight
operand at all), the output projection, then the second window's copy of the
same pair, then the MLP phases and the M64 N256 K128 merge. The weight extents
it reports agree with the harness dump to within a 12-byte unpadded tail
(54548 against 54560 for enc0), and the widest phase gives each layer's
patch-merge or patch-expand width.
D4R_CUDA_LAUNCH_STATS=1 synchronizes after and dumps every argument of every
launch, which costs about 1.2 GB and 5 s per frame - fine for a one-off
inventory, too much for the edit/measure loop of a single kernel.
D4R_CUDA_LAUNCH_STATS_FILTER=<substring> keeps only the kernels whose name
contains it, and is checked before the synchronize so the rest of the frame is
neither waited on nor written.
Verified: with the filter set to swin_enc0 the dump is 54 files / 839 MB
instead of 314 / 3.7 GB, the run takes 5 s instead of 17, and the 54 argument
files are byte-identical to the ones the unfiltered run produced.
The bridge's other instrumentation - D4R_CUDA_LAUNCH_STATS with
D4R_CUDA_LAUNCH_DUMP_DIR - already dumps every allocation, surface object and
texture object named in a launch's packed argument buffer, after the launch has
finished. That is enough to check one reimplemented kernel on its own: the
translated kernel's surface files are its output, and a native replacement has
to reproduce them.
What was missing was the comparison. compare_launch_dump.py keys the files by
kernel, argument offset, kind, shape and format rather than by the global launch
sequence, so a replacement that adds launches of its own does not break the
alignment, and it reports the first differing byte per argument.
Verified end to end today: cuda_dldn_engine_swin_enc0_kernel's PTX module was
compiled offline with d4r_emit, served from a native directory (the bridge logs
`[d4r-native] "cuda_dldn_engine_swin_enc0_kernel" -> ...hsaco`), and its run is
byte-identical to the translated one across all 54 of its dumped arguments -
weight image, activation buffer, four output surfaces. That also shows the
offline build reproduces the loader's own compile exactly, which every native
denoiser layer will rest on.
docs/native-kernels.md gains the dump-and-diff step, which is the only way to
localise a mismatch to one of the eleven layers that sample textures and write
surfaces.
A replay only means something if the denoiser gets the bytes the game gave it.
The capture now records, for every bound guide, the caller's own DXGI format and
texel size next to the plane (they differ from the CUDA array's for every
converted format, and B8G8R8A8_UNORM is four bytes to D3D12 and sixteen to the
float array), the names the caller registered each surface under as well as the
names the denoiser reads it as, the exposure texture's own geometry, and the
feature's creation state - dimensions, quality, create flags, output-subrect
setting and the effective render preset. A frame whose staging is not readable
aborts its whole capture instead of writing a descriptor without its plane.
The harness reads all of it back: it sizes and creates the feature from the
capture, creates each guide in the recorded format at the recorded size, uploads
the recorded bytes, applies the recorded scalars, subrect origins, inversion
flags and camera matrices, binds the exact source names and clears the ones the
frame stopped registering. A missing guide descriptor, a missing or mis-sized
plane, a truncated parameter line, a claimed exposure with no geometry, or a
capture taken before the creation state was recorded is refused rather than
filled in with a default, because a replay that invents any of it still renders
a plausible image that matches nothing the game produced.
Ray Reconstruction's eleven DLSS-D Swin layers still run ZLUDA's compile of
NVIDIA's PTX; at 1280x720 -> 3840x2160 on gfx1201 they are 30.2 of a 31.6 ms
frame. The denoiser's high-fidelity output kernel is the first one with a native
replacement: make_ptx.py replaces its whole-texel RGBA16F store
(sust.b.2d.v4.b16.zero) through d4r_sust_v4b16, kernels/build.sh builds the pair
as the `rr` family, and packaging builds it into every target folder including
the RDNA4 FP8 ones.
Both libraries define modules named cuda_dldn_engine_*, with different text, so
one manifest cannot take the union of their hashes: kernel_manifest.py now takes
the denoiser DLL as --rr-dll and lets only its hashes authorize a
cuda_dldn_engine_* binary, and package_release.sh passes it when the denoiser is
bundled.
Verified in the D3D12 harness with Ray Reconstruction at 1280x720 -> 3840x2160:
the output kernel drops from 1.44 to 1.00 ms/frame and the 66 MB raw output is
byte-identical to the run with every kernel translated.
docs/native-kernels.md gains the denoiser family, the per-kernel cost table and
the instruction counts that show why the Swin layers are the remaining work.
Run the signed CUDA-capable denoiser through feature 13 with independent D/E presets, pre-Init discovery, per-frame guide and camera snapshots, normalized integer guide staging, paired alpha output, and recorded-resource retirement.
Add the required ZLUDA/LLVM patch, behavioral configuration tests, GPU harness coverage, optional denoiser installation and packaging, and tested support limitations. Preserve upstream attribution and distinguish LLVM licensing from the ZLUDA portions of the patch.
scripts/check_redistributable.py refuses NVIDIA binaries (by name and
version resource), NGX SDK headers, PTX and code objects, PTX translations
(marked by ptx2glsl.py/condense.py) and the engine's model data in the
repository, and in each release variant (clean/installer/full) whatever
that variant does not contain. package_release.sh runs it before zipping;
scripts/git-hooks/pre-commit runs it locally.
Signed-off-by: woutermans <[email protected]>
A standalone Vulkan inference engine (engine/) for DLSS presets K and M that
runs from weights extracted from the user's own nvngx_dlss.dll, with d4r's
own input, network and reconstruction shaders or the native HIP network
layers, recorded into the game's command list by the shim. Opt-in
([Engine] Enabled / D4R_ENGINE=1); anything uncovered falls back to the
CUDA/ZLUDA path with the reason logged.
Byte-identical to NGX's CUDA backend under the phase-capable reference
(1706x960 -> 2560x1440, native HIP network, eleven frames, RX 9070 XT):
- K render- and display-resolution motion vectors, HDR and LDR
- K game-supplied exposure and automatic exposure with any pre-exposure
and exposure scale (one residual: effective exposure exactly 2 or 4)
- K and M with the DLSS indicator's inverted-axis parameters
- M with any exposure settings (NGX ignores them for M)
K packages need the current compile_k.py (marker D4RO0003).
Also: HIP shared buffers are pooled across feature recreation (ROCm leaks
destroyed imports), hipnet.bin must match the preset, M-only games do not
load the K model, install_d4r_runtime.sh stages k, k-ldr, m, m-ldr with
hip/, the activation table has a generator (gen_activation_m.py), and
package_release.sh can bundle the engine's shaders (never its weights).
Signed-off-by: woutermans <[email protected]>
Implement NGX's Vulkan API over the existing CUDA backend. Compute shaders
pack the game's sampled images into buffers shared with CUDA (device-local
memory exported through the Wine bridge), and a device event in the game's
command buffer holds its queue until the result is written back, keeping
frame age at zero. DLSS reads inputs through pitch-linear textures and its
native output kernel writes the shared buffer directly; input and
completion waits are adaptive.
Make it work across engines: shared buffers are concurrent over every
compute-capable queue family (Indiana Jones records DLSS on async compute),
depth kept in an R32F colour image is accepted under a depth aspect, and the
extension-requirement query allows a null list. Spare buffer sets are
imported on the first frame and never while a frame is pending, since an
import can block on GPU work that is waiting for us; host staging covers any
shortfall.
Fix two exit hazards. A game quitting without NGX shutdown left the GPU
waiting on an event that only a killed thread would set, resetting the ring
10 s later: the bridge now runs a shim hook before teardown and skips all
CUDA destroy/sync calls once Wine is shutting the process down. The shim's
own entry point skips CRT static destructors at process termination, where
locks held by killed threads made the exit spin forever.
Add a Vulkan DLSS harness (dynamic, recreate, discard, invalid, exit,
compute-queue and in-flight modes), staging/depth probes, ABI/config tests,
and an OptiScaler settings overlay for Vulkan games.
NVIDIA's enc1 and enc2 clamp the patch merge's result to ±2π (as f16) before
its e4m3 encoding (max.f16x2 / min.f16x2 with 0fC0C90FDB / 0f40C90FDB after the
merge's last MMA). The native layers did not, in either template or set:
merge_clamp (swin_common.h) now does, at the three sites that encode the merged
tokens (swin_block.h's e4m3-activation and f16 paths, swin_tok.h), and
swin_check.py's model clamps the same way.
Measured on an RX 9070 XT against a run with no native kernels (preset L,
accuracy set, the flags of Ghost of Tsushima, 1280x720 -> 3840x2160, one
frame): native enc1 alone 34 dB and enc2 alone 31 dB from the translated
kernels before, byte-identical after; the whole native set 28 -> 80 dB (the
rest is dec1, 0.09% of the output values). Preset M's fast set against
translated M: 30.8 -> 35.1 dB after 30 frames. The gfx11 path has the same
change and was not run on hardware.
The repo's tests pass (36, 3 skipped).
L's unfolded enc0 and dec0 are still ZLUDA's compile of NVIDIA's PTX and cost
2.95 ms of the preset's 5.86 ms per frame (RX 9070 XT, 1280x720 -> 4K). The
embedding, the patch expand, the Swin block (32 channels, 2 heads, at pixel
resolution) and the patch merge are now native token-lane code; the texture
work around them stays ZLUDA's compile.
enc0l_tail.hip cuts in at the barrier after the feature staging and dec0l_block.hip
before the output code that reads the block's f16 result out of the last MMAs; make_ptx.py
splices either in when D4R_TEX_SRC names it and build.sh selects both for the gfx12
native-FP8 builds. swin_tex.h carries the indexing the two tails share; swin_tok.h gains
the pixel-order token layout (SWIN_TOK_NATURAL), ZLUDA's attention key order
(SWIN_PV_ZLUDA_ORDER) and swin_tok_core, which the encoded tests reuse. swin_common.h
takes the block weight layout (moved out of swin_block.h) and the exact and approximate
reciprocal / mix forms the tails need without the HIP headers (D4R_NO_HIP_RUNTIME).
The f32 sums stay in ZLUDA's order (K slots of the FP8 WMMAs, key order of the f16
attention product) and rsqrt/rcp use the hardware approximations ZLUDA lowers to, so the
harness's final frame is byte-identical to the translated kernels: all four enc0 flag
variants, 720p and 1080p render, 1440p and 4K output, static and moving scenes, 100
frames, and the accuracy set. With M's own sum order about one token in 20,000 differed.
M's layers compile to the same instructions and its frame is byte-identical.
enc0 1.69 -> 0.63 ms and dec0 1.26 -> 0.54 ms at 720p->4K, 2.73 -> 1.16 and 2.28 -> 1.01
at 1080p->4K; the preset 5.83 -> 4.06 and 8.63 -> 5.69 ms. -DD4R_SWIN_K32_MIX rounds the
f16 products earlier for 0.56 and 0.49 ms and is not byte-identical, so it is not built by
default. The repo's tests pass (36, 3 skipped).
Shim
- Adaptive output-event wait: sleep to ~120 us before the predicted
completion (smoothed per wait site), spin near it, back off when late.
D4R_SHIM_OUTPUT_POLL_US restores a fixed interval.
- Defer the input-marker wait into the worker thread with VRAM interop
(D4R_SHIM_DEFER_MARKER, default on), taking the host setup off the
critical path.
MicroCUDA
- Experimental ZLUDA-free backend in wine_nvcuda_bridge, plus
tools/microcuda and scripts/build_microcuda.sh. Serves accuracy, fast
and native kernels; unregistered modules fall back to the reference
path. Unclassified writes invalidate prepared weights.
M kernels
- SWIN_PREP_IDENTITY_AT: key weight prep by allocation identity
(MicroCUDA); enc3 tube uses it with 8 slots. ZLUDA ignores it.
- dec2 (gfx1101): SWIN_PACKED_RESIDUAL, byte-identical, ~1.6% faster.
- Opt-in gfx12 hardware fp8 conversion (SWIN_FP8_HWCVT), unverified
on hardware.
Native K translators
- ctile: handle variants that re-materialize the parameter base (mvlo).
- rewrite_output: check live-in registers, not just outputs.
- build.sh: native input covers all 8 flag variants, output the 4 HDR
"max" variants; depth-tile skip stays mvhi-only.
Tools
- dump_runner and bench_dlss_kernel harness additions.
Alan Wake 2 uses the format for depth when enabling Ray Reconstruction.
The CPU path converts half to float in convert_row_in.
The VRAM interop path blits R16F/R16UNORM depth to an R32F conversion image before copying into the canonical buffer.
Signed-off-by: Tk-Glitch <[email protected]>
- Add docs/in-place-inputs.md: reading the DLSS inputs in place does not work,
because a hardware texture fetch on a HIP texture object over memory imported
from vkd3d-proton returns stale data while plain loads and cuMemcpy2D are
correct; records the measurements, the rule-outs and the reworked-fetch
workaround that turned out not to pay for itself.
- Add tools/hip_vk_texture_stale_repro.cpp and its build script, the native
Vulkan + HIP reproducer that rules out a general ROCm or RADV bug.
- Tag redirected surfaces as R10G10B10A2 ('R3') as well as RGBA16F ('R2'), and
let the native output kernel store the game's R10G10B10A2 texels itself.
- Import linear, exportable game textures (vkd3d-proton patch 0003) into HIP
so the output kernel writes the game's output texture with no game-side copy.
- Add ID3D12DXVKInteropDeviceD4R3 with GetVulkanLinearImageInfo.
- Gate both paths behind D4R_SHIM_OUTPUT_PACKED and D4R_SHIM_INPLACE, and use
them only for split frames of preset M whose output kernel honours the redirect.
- Convert on the host when a recorded packed frame finds the wrong kernel.
- Give the harness an R10G10B10A2 output, and add D4R_SHIM_SPIN_POLL so
frame timing is not quantised to the poll sleep.
- Document the measured removal of the output copies in performance.md.
make_ptx.py turns the 32 cvt.rn.f16x2.e4m3x2 of rrlite_dec0_4x4_folded into calls of d4r_dec_e4m3x2 (tex_common.h) when D4R_TEX_FP8=1: v_cvt_pk_f32_fp8 and two conversions to f16 instead of ZLUDA's integer lowering (about 25 instructions each; the kernel goes from 4058 to 3418 instructions). The conversion equals ZLUDA's e4m3_to_f16_bits for every code except the NaN codes 0x7f / 0xff, checked over all 65536 code pairs on an RX 9070 XT, and the head never stages those codes (it clamps to +-448 before encoding). Harness frames are byte-identical in the accuracy and the fast set, over a 16-frame moving sequence in the accuracy set. dec0 at 2227x1253 render: 0.43 -> 0.38 ms (accuracy), 0.36 -> 0.30 ms (fast). -DD4R_TEX_SOFT_E4M3 gives ZLUDA's lowering; builds without D4R_TEX_FP8 keep the PTX instruction.
On the gfx12 native-FP8 path the enc0 tail and the dec0 head encode their results with v_cvt_pk_fp8_f32 on the f32 copies of the f16 values (e4m3x4_f32 in tex_common.h) instead of ZLUDA's integer e4m3x2. The clamps keep the conversion in range: +-2pi for enc0 (the f16 bound hi[0], same in f32) and +-448 for dec0, both as a median of three, where a NaN gives the lower bound. The codes equal e4m3x2 for every non-NaN f16, checked over all 65536 patterns on an RX 9070 XT; a NaN in dec0 now encodes as -448 instead of the NaN code. Harness frames are byte-identical in both sets (LDR and HDR, motion vectors at render and display resolution). dec0: 0.21 -> 0.15 ms at 1280x720 render and 0.74 -> 0.53 ms at 2560x1440; enc0 changes by 1-2% (its time is ZLUDA's compile of the feature code). -DD4R_TEX_SOFT_E4M3 (D4R_TEX_CFLAGS) keeps e4m3x2.
The remaining two layers (96 channels, four heads) use the same swin_tok.h template: byte-identical on the same recorded launches and harness sequences, enc2 0.22 -> 0.14 ms and dec2 0.20 -> 0.14 ms at 1280x720 render, 0.75 -> 0.47 and 0.69 -> 0.47 ms at 2560x1440. Skipping windows that lie entirely outside the image is also exact but below the measurement noise (at most 2.4% of the windows of every other tube launch), so it is not kept; true16 code generation remains not tried.
woutermans