rr: add FSR Ray Regeneration, exact RDNA3 inference and motion experiments

Add an experimental FSR Ray Regeneration backend for D3D12 RR requests:
- Convert depth, motion, normals and albedos into AMD denoiser inputs.
- Split and recompose diffuse/specular signals, then run Super Sampling.
- Add stability/detail controls, demodulation guards and halo limiting.
- Handle AMD provider discovery under NVIDIA GPU spoofing.
- Add runtime probes, synthetic reference scenes and per-dispatch profiling.

Add exact gfx11 DLSS-RR inference using f16 cooperative matrices:
- Reproduce FP8 conversions, accumulation and camera reprojection arithmetic.
- Widen model weights and avoid requiring VK_EXT_shader_float8.
- Extend model building, replay tools and descriptor metadata.

Expand motion preprocessing and experimental AW2 native-motion support:
- Add tiled LDS dilation, padded allocation handling and thin-only sky fill.
- Order dilation with input waits and NGX on the selected stream.
- Add jitter-aligned native composition, hybrid routing and wire history repair.
- Add prepared masks, sparse repair and independent motion/repair controls.

Add optional fused R11G11B10 input/output conversion, sampled stage timings
and profile summarization. Extend configuration, packaging, documentation,
FidelityFX headers and CPU/GPU regression coverage.
This commit is contained in:
ayo committed 2026-10-16 04:55:25 -04:00
1 parent 17c1828f97
commit 34e6b0fe71
83 files changed
+11300 -195

No files matched your search

+1 -1
View File
@@ -112,7 +112,7 @@ OptiScaler 0.9.4 checks a relative `NvngxPath` against the process working direc
## Logs
The launcher writes each run's logs to `~/.cache/d4r-dlss-captures/game-<exe>-<time>/`. `d4r_nvngx.log` holds the shim's log. With `[Debug] Profile = true` it also records one `D4R_PROFILE` line per frame, with the stage timings.
The launcher writes each run's logs to `~/.cache/d4r-dlss-captures/game-<exe>-<time>/`. `d4r_nvngx.log` holds the shim's log. With `[Debug] Profile = true` it measures stage timings every frame and records sampled `D4R_PROFILE` lines. `ProfileLogEvery = 60` (the default) logs the first five frames and then every 60th frame; set it to `1` to log every frame, or another positive integer for a different interval. Restart the game after changing it. `D4R_PROFILE_LOG_EVERY` overrides the INI setting. FSR RR stage summaries average that many completed GPU samples; native Vulkan profiling uses the same log interval. Sampling reduces log writes while retaining profiling measurements and per-frame throughput intervals. `D4R_PROFILE_LOG=0` suppresses timing rows and FSR RR summaries.
Output completion uses event queries with 200 us NtDelayExecution sleeps by default. `D4R_SHIM_BLOCKING_SYNC = 0`
under `[Env]` restores context synchronization for an A/B check. The profile records
+103
View File
@@ -0,0 +1,103 @@
# FSR Ray Regeneration as the Ray Reconstruction backend (experimental)
`[RayReconstruction] Backend = fsr` answers a game's DLSS Ray Reconstruction request (NGX feature 13)
with AMD's FSR Ray Regeneration 1.2 instead of NVIDIA's denoiser. `Backend = dlss` (the default) keeps
d4r's existing DLSS Ray Reconstruction. Changing the backend needs a game restart, or the game
recreating its Ray Reconstruction feature.
Status: works on an RX 7700 XT (gfx1101) under Proton with the synthetic test scene described below.
Alan Wake 2 logs from 2026-10-14 confirm denoiser dispatch and internal DLSS K upscaling, but the
game test reports excessive noise and artifacts. **Game image quality is not validated.**
AMD supports FSR Ray Regeneration only on RX 9000 / Windows 11. This
route is Linux/Proton only, and nothing here is validated on native Windows or for native Vulkan games.
## What runs
In the game's own command list and frame:
1. `kernels/support/fsr_rr_prepare.comp` (Vulkan, through vkd3d-proton's interop) turns the DLSS-RR
inputs into the denoiser's inputs:
- signed linear depth (from hardware or linear depth and the game's ViewToClip);
- previous-minus-current UV motion with the camera's view-z delta;
- octahedral normals plus roughness;
- linear albedos.
It splits the noisy colour into a diffuse and a specular signal by albedo and demodulates both
(after Zach Hembree's and burak113's GPL OptiScaler adapter). What cannot be demodulated, the sky,
and particles (when `DLSSD.ColorBeforeParticles` is given) go to a residual that bypasses the
denoiser.
2. AMD's `amd_fidelityfx_denoiser_dx12.dll`, unmodified, runs through `ffxDispatch`. Each lobe uses
the indirect path only when the game supplies that lobe's hit distance; otherwise it uses the
direct path.
3. `fsr_rr_compose.comp` remodulates the denoised signals and adds the residual.
4. An internal Super Sampling feature, configured like the user's own (`[DLSS] Model`, engine or
CUDA), upscales to the output size the game asked for.
`nvngx_dlssd.dll` is not used. When a frame cannot be denoised (missing guides or matrices, jittered
motion vectors, `DLSSD.OutputAlpha` requested), the evaluation fails and the reason is logged. It is
never upscaled noisy as if it had been denoised.
## Requirements
- AMD's `amd_fidelityfx_denoiser_dx12.dll` from FidelityFX SDK 2.3. Tested with SHA-256
`48f1e5888ba6a0a3d59a98b9751e37c392b0f7b8c223d0082d5c1f40642879d3`, file version 1.2.0.2740.
Put it in the d4r folder, or set `[RayReconstruction] FsrDenoiser`. It is AMD's binary under
AMD's terms, not part of this repository.
- d4r's vkd3d-proton built with `D4R_VKD3D_FSR_RR=1 scripts/build_vkd3d_proton_d4r.sh OUT`. This build
adds upstream dxil-spirv's accepted scalar FP8 fallback (PR 307) and extended emulation.
- `DXIL_SPIRV_CONFIG=wmma_rdna3_workaround` in the game's environment.
- Proton's `amdxc64.dll`. vkd3d-proton loads it only when the GPU is reported as AMD, so under
`PROTON_FORCE_NVAPI` the shim loads it after confirming through sysfs that the GPU is AMD's.
Without it, AMD's runtime offers no provider.
## Tuning
Two `[RayReconstruction]` keys adjust the denoiser. Both are off by default.
- `FsrStabilityBias` (`D4R_FSR_RR_STABILITY_BIAS`): AMD's temporal stability bias. `auto` keeps AMD's
default of 1. With that default, reflections and fine detail come out blurred. On Alan Wake 2's water,
the result had about half the detail of DLSS Ray Reconstruction. `0` brought the replayed water capture
from 29.5 to 36.6 dB PSNR against DLSS RR's output. On the synthetic scene it roughly halved the error
against ground truth, whether the camera was still, moving or just after a cut. GPU time was unchanged.
- `FsrDetail` (`D4R_FSR_RR_DETAIL`, 0–1): the share of the colour that skips the denoiser and goes
through the residual. It keeps detail the denoiser removes as noise: hair, water ripples and foliage.
The cost is some grain. With `0.25` on top of stability bias 0, PSNR against DLSS Ray Reconstruction's
output went up by 0.4 dB on the water capture, 1.4 dB on the street capture and 0.8 dB on a walking
capture in town. Frame-to-frame flicker off the water went from 0.0069 to 0.0073 (DLSS RR: 0.0060).
Two guards are on by default, because a game's single colour has light in it that the albedos did not
shape, such as fog, sky light, or the background behind a thin object's edge:
- No albedo below 0.1 divides a signal (`D4R_FSR_RR_DEMOD_FLOOR`). Dividing fog by dark foliage
overflowed, and the fog lost its colour and haze.
- Each denoised lobe is capped at the brightest noisy input of that lobe in its 3×3 neighbourhood
(`D4R_FSR_RR_HALO_LIMIT`, 0 turns it off). Without the cap, the denoiser leaks a thin object's large
demodulated signal across its edge, which shows as bright or blue fringes around poles, wires, hair
and the edges of water.
`D4R_FSR_RR_KEY_<n>=<value>` overrides any of AMD's six configuration keys, for diagnostics. The log
lists AMD's defaults when the denoiser context is created.
## Known limitations
- One noisy colour cannot be split exactly into diffuse and specular. On the test scene this route
flattens some shading gradients compared with DLSS Ray Reconstruction.
- Normals are assumed to be in world space. The depth delta comes from the camera's motion only, so
moving objects get no depth-change term.
- Guide subrect origins, jittered motion vectors and alpha output are not supported.
- Cost scales with render resolution. AMD's denoiser alone, on gfx1101:
- about 4 ms at 960×540 (2 signals);
- about 8 ms at 1280×720;
- about 26 ms at 1920×1080 (4 signals).
## Checks
- `tools/fsr_rr_probe.cpp` (`scripts/run_fsr_rr_probe_proton.sh`) tests AMD's runtime alone: provider,
context, multi-frame denoising measured against ground truth, and reset.
- `tools/rr_backend_check.cpp` (`scripts/run_rr_backend_check_proton.sh`) drives a feature-13 request
through the shim and compares the output with the scene rendered noise-free at output resolution.
`--feature 1` gives the Super-Sampling-only baseline. With `D4R_PROFILE=1`, the log reports GPU time
per stage.
- `--hw-depth --separate-eval-params` checks callers that put depth/roughness modes only on the
creation parameter object. The FSR adapter uses the resolved creation/per-frame settings; it
must not silently interpret hardware depth as linear when a frame omits the creation setting.
+172 -2
View File
@@ -74,9 +74,19 @@ Initial validation: a gfx1101 fast L build completed two Ultra Performance harne
**Ray Reconstruction, the denoiser (`kernels/tex` + `cuda_dldn_engine_*`).**
- **Network:** `nvngx_dlssd.dll` defines the DLSS-D denoiser: six Swin encoders (`cuda_dldn_engine_swin_enc0_kernel` … `enc5`), five decoders (`dec0` … `dec4`), a high-fidelity kernel-prediction network whose output kernel is `cuda_dldn_engine_hkpn_output_kernel_transformer`, and the luma/auto-exposure/`reduce_sum` helpers. It shares no kernel with the Super Resolution libraries.
- **Native today:** the hkpn output kernel (and its `_diamond_wallaby` twin), built by `kernels/build.sh rr` as a wave64 surface-store replacement. `make_ptx.py` replaces `sust.b.2d.v4.b16.zero` — a whole RGBA16F texel — with `d4r_sust_v4b16`; the neural body and the texture reads are unchanged.
- **Still translated:** all eleven Swin layers. ZLUDA compiles NVIDIA's PTX at load time; the table below measures the resulting kernels' GPU execution, not JIT latency. A native replacement must preserve each layer's attention, norm and merge stages, as `kernels/m/rrlite_*` does for the upscaler.
- **CUDA route:** all eleven Swin layers remain translated. ZLUDA compiles NVIDIA's PTX at load time; the table below measures the resulting kernels' GPU execution, not JIT latency. A native replacement must preserve each layer's attention, norm and merge stages, as `kernels/m/rrlite_*` does for the upscaler.
- **Manifest:** the denoiser DLL is passed to `kernel_manifest.py` as `--rr-dll`, because its hashes may only authorize `cuda_dldn_engine_*` replacements. Without that split, a name both libraries define — `dl4rt_input_kernel` — would let a super-resolution binary be served for the denoiser's different implementation of it.
The separate [Vulkan RR engine](../engine/rr/README.md) also has an exact RDNA3 build: `D4R_RR_GFX11=1
D4R_RR_RELAXED=0 engine/rr/build_native_model.sh ...`. It runs the eleven native network stages with f16 cooperative
matrices and reproduces ZLUDA's FP8 lowering, without `VK_EXT_shader_float8`. On an RX 7700 XT, preset D, AW2 town
replay at 853×480 → 2560×1440, two engine runs and the CUDA reference are byte-identical over all 24 RGBA16F frames.
Replay GPU time is about 11.4 ms versus 27.9 ms for the current CUDA control; these are replay measurements.
Live AW2 denoiser GPU time was about 27.3 → 12.0 ms; whole-frame intervals were 42.6 → 24.4 ms in the faster
sessions and 53.4 → 36.2 ms in the slower sessions (one launch per backend in each state). The slower game state
remains; these figures are not pooled across states. The model's weights stay local. Select `[RayReconstruction] Backend = dlss` and `[Engine] Enabled = true` with the
model in `d4r/engine/rr-d`; FSR RR retains its own route. Preset E and other RDNA3 GPUs remain unverified.
Measured on the RX 9070 XT (gfx1201, RDNA4), Ray Reconstruction preset E,
1280×720 → 3840×2160, with `D4R_CUDA_KERNEL_PROFILE` in the D3D12 harness:
The earlier 20-frame run excluded the first 6 frames, with native hkpn output
@@ -295,6 +305,166 @@ Runs of the same build vary by about ±3% per kernel. M additionally skips its o
`[Interop] MotionVectorDilation = 1` or `2` (`D4R_MOTION_DILATION`) extends nearby foreground motion into pixels whose depth otherwise belongs to the background. The value selects the opaque-surface radius in render pixels. Both enabled values also fill sky gaps using the nearest foreground pixel within an eight-render-pixel circle (far depth 0 for reversed Z, 1 for normal Z). This is the wider sky fill from the comparison; it can strengthen trails in some frames and remains opt-in through `MotionVectorDilation`. This is a depth-guided approximation to better thin-feature motion coverage, not engine-side conservative rasterization. It improved the captured Townfall rooftop where the earlier native-encoder fallback did not resolve the artifact.
The d4r-owned GPU kernel preserves the original depth/motion buffers and writes a separate RG16F velocity texture. The worker waits for its completion before NGX evaluates. No per-frame CPU readback is needed. It requires same-frame linear VRAM inputs, full-resolution unjittered motion vectors and zero subrect bases. Unsupported features or unavailable kernels keep the original vectors. The default is `0` (off); larger radii can affect disocclusion and unrelated foreground edges. The method does not recreate geometry missing from every input frame or establish RTX parity.
The d4r-owned GPU kernel preserves the original depth/motion buffers and writes a separate RG16F velocity texture. The input marker precedes dilation, and dilation precedes NGX on the same bridge-selected stream (including the optional priority stream); there is no dilation-specific CPU completion wait. No per-frame CPU readback is needed. It requires same-frame linear VRAM inputs, render- or display-resolution unjittered motion vectors and zero subrect bases. Unsupported features or unavailable kernels keep the original vectors. The default is `0` (off); larger radii can affect disocclusion and unrelated foreground edges. The method does not recreate geometry missing from every input frame or establish RTX parity.
`scripts/build_wine_nvcuda_bridge.sh` embeds the support kernels when `D4R_ROCM_DIR` points to ROCm clang. All six gfx110x/gfx120x targets compile; physical validation is currently gfx1101 only. A bridge built without the support kernels reports the feature unavailable rather than changing inputs. Optional GPU regressions cover forward/reversed depth, two radii, jitter, padded rows and preservation of input buffers; see `tests/test_motion_kernel_gpu.py`.
For equal motion/render grids, zero jitter and radius 1 or 2, the bridge selects
`d4r_motion_dilate_lds`: 16x16 output tiles use a 32x32 depth halo and wave32 row
foreground masks. The nearest sky fill preserves the generic kernel's exact
(distance squared, x, y) tie-break. The additional padded export keeps the original
72-byte parameter structure and passes allocation height separately; inactive
rows/columns, including row-pitch padding, copy through from the original motion
buffer in the same launch. Display-resolution or jitter-mapped grids retain
`d4r_motion_dilate` and its existing padding-copy fallback.
Deferred CPU marker polling remains enabled with dilation. GPU input waits can
also run with the LDS path once that slot has successfully allocated its output
texture and submitted a dilation through the new bridge. Cold slots, changed
allocation geometry, older bridges and the generic path use CPU marker polling
so lazy allocation/module loading or synchronous fallback copies cannot block
behind an unreleased GPU wait. Existing RR auxiliary-upload and staged-input
synchronizations remain in place.
`motion_dilation_wall` measures host preparation/submission time after this
change, including allocation on cold slots; it no longer waits for the kernel to
finish. Compare pipeline/throughput fields for end-to-end cost. On the GPU-wait
route, the evaluation event precedes the input wait and dilation, so
`dlss_gpu` includes dilation after subtracting input readiness; on the CPU-polled
route its start event follows dilation. Those GPU timing scopes are different.
GPU regression tests compare both kernels byte-for-byte on deterministic random
inputs and independently check image/block-edge sky ties, both depth directions,
both radii, padded rows/bottom rows, guards and immutable inputs.
### Opt-in thin-only sky fill
`[Interop] MotionDilationThinSky = true` (`D4R_MOTION_THIN_SKY=1`) selects
`d4r_motion_dilate_lds_thin` when dilation is enabled and the equal-grid, zero-jitter
LDS route is available. The default is false and the existing nearest-foreground
behavior remains unchanged. Opaque radius-1/radius-2 dilation is unchanged.
The sky search accepts foreground runs of at most three pixels along **either**
axis. This retains long horizontal wires and vertical poles; requiring short runs
in both axes would discard them. A three-pixel mask halo beyond the search tile
prevents workgroup boundaries from truncating run classification. Nearest-distance
and coordinate tie-breaking remain the same among eligible candidates. Mapped-grid
or older-bridge routes retain nearest fill and report that thin fill is unsupported.
Both sky-fill modes currently assume normalized depth with sky at 0 (reversed) or
1 (forward). The available Alan Wake 2 RR captures have forward depth greater than
1 throughout; they contain no foreground according to this existing threshold and
cannot validate a sky-fill quality benefit. Thin mode is an experimental hypothesis,
not a measured in-game quality improvement. It stays opt-in; no automatic render-ratio
policy has been enabled. Restart after changing this setting.
### Experimental AW2 native motion coverage
The private AW2 driver prototype can feed output-resolution raster motion to
NGX with `D4R_NATIVE_MV_LIVE=1`. Its live path also uses a thin-silhouette halo:
`D4R_NATIVE_MV_THIN_HALO=0` selects the original 3x3 native compositor instead.
The halo extends only sky pixels toward foreground runs shorter than five output
pixels on at least one axis. Its radius tracks two render pixels, capped at six
output pixels. Thick surface interiors do not receive that extra expansion.
The original compositor and 80-byte parameter ABI remain available; the Wine
bridge exports the additional `d4rComposeNativeMotionThin` operation.
This requires the private native-raster driver and GPU-ordered owned-input
validity contract. It is experimental, defaults remain unchanged, and this is not
a general engine-independent native-velocity feature. A 24-frame AW2 forward-walk
experiment at 853x480 -> 2560x1440 reduces paired wire dropout by 67.5%, beading by
22.7%, temporal contrast error by 23.2%, and nearby darkening proxy by 39.9% after
eight warm-up frames. These are one-scene proxies, without native-color ground
truth, and do not establish artifact-free rendering. On gfx1101 the complete
halo compositor takes 1.175 ms kernel-only versus 0.148 ms for the original
native compositor; no matched live performance gain is claimed.
The private AW2 path also supports `D4R_NATIVE_MV_HYBRID=1`: retain native
motion within six output pixels of a thin native silhouette and preserve the
game's nearest-sampled motion elsewhere. This avoids substituting the complete
native geometry/camera field over distant surfaces. It requires the additive
`d4rComposeNativeMotionHybrid` bridge operation and matching low-resolution motion
dimensions; unsupported layouts retain the full native fallback. It defaults off.
`D4R_NATIVE_MV_WIRE_SKY_ONLY=1` restricts color repair to silhouettes with sky on
both sides of the thin axis, also default off. Fractional coverage also checks the
nearest native depth: a different foreground surface or NaN depth is preserved,
rather than receiving color from an adjacent wire. Depth-matched wire centers and
sky remain eligible. These are experimental controls,
not an artifact-free rendering or 2.47 ms performance claim.
Wire-color repair is independent of the extra motion halo. With a matching
bridge, `D4R_NATIVE_MV_THIN_HALO=0` selects jitter-aligned 3x3 native composition
while leaving `D4R_NATIVE_MV_WIRE_HISTORY` active. Earlier builds gated both
operations on the halo flag, which prevented an isolated halo-off comparison.
All these environment controls belong in the per-game INI's `[Env]` section.
`D4R_NATIVE_MV_SUBSTITUTE=0` decouples wire repair from NGX motion substitution.
With `D4R_NATIVE_MV_LIVE=1` and `D4R_NATIVE_MV_WIRE_HISTORY=1`, the native field
is composed only for repair; NGX retains the game's vectors, creation flags and
MV scale. Dilation still follows its existing setting; use
`D4R_MOTION_DILATION=0` for an unmodified game-vector comparison. `SUBSTITUTE`
defaults to 1, preserving the combined path. `LIVE=0` still bypasses both native
composition and repair. This is an experimental isolation control: repair-only
can still alter distant thin silhouettes and has not established visual stability.
Keep CPU input waiting for these comparisons; no GPU-wait safety gate is removed.
Profile lines also identify the feature, native-field readiness, actual native
substitution, requested repair and requested GPU completion-word path. Use
`scripts/summarize_d4r_profile.py LOG` to group warm samples by feature instance,
resolution, model, quality, flags and routing. Missing GPU timings remain missing;
CPU marker/pipeline times and delayed Vulkan samples are summarized independently.
With profiling enabled, sampled live-native frames also report
`native_motion_gpu` and `wire_repair_gpu`. These bracket the composition kernel
and color-repair dispatches on the NGX stream; elapsed times are read after the
existing publication synchronization. They add no CPU readiness wait and remain
`-1` when inactive or unavailable. They describe the current frame, while the
existing game-queue Vulkan timestamps are read from older completed slots.
`D4R_NATIVE_MV_PREPARED=1` selects an exact implementation of the existing full
native-motion and wire-repair policy. A 32x32-tile preparation kernel builds a
packed native silhouette mask and per-pixel wire axes once; a wave64 kernel
composes the same motion field. Sparse repair consumes those axes instead of
repeating the depth classification. RGB/depth history entries marked age 255
are left unspecified because the consumer checks age before reading them;
rendered RGBA and readable history remain unchanged. The additive bridge
operations are `d4rComposeNativeMotionPrepared` and
`d4rStabilizeThinWiresPrepared`; missing exports, incompatible flags/layouts or
allocation failure retain the earlier implementation. The generic default is
0; enable it under `[Env]` and restart for an A/B test. This is a performance
implementation switch and does not enable hybrid motion or sky-only repair.
On gfx1101, matched replay tests measured roughly 0.28 ms for native motion plus
shared preparation and 0.17 ms for sparse repair/apply, excluding NGX. These are
kernel replay measurements, not a live total or a claim that distant artifacts
are resolved. The sampled `native_motion_gpu` field includes the shared wire
preparation when this route is active.
The prepared compositor also remains available with `WIRE_HISTORY=0`; turning
color repair off does not require the slower legacy native compositor.
### Optional R11G11B10 interop conversion
`D4R_SHIM_FUSE_R11_INPUT=1` samples an R11G11B10 color image directly into the
pitched RGBA16F input buffer. `D4R_SHIM_FUSE_R11_OUTPUT=1` writes the RGBA16F result
buffer directly into an R11G11B10 storage image. Both default off. These reuse the
Vulkan staging shaders to remove intermediate-image copies and blits; NGX model,
input readiness, frame age and temporal state are unchanged. Unsupported resource
usage or descriptor allocation failure retains the original transfer path. Output
requires `ALLOW_UNORDERED_ACCESS`; input requires shader-resource access.
Descriptors and owner images remain alive through recorded allocator retirement,
with bounded retention for changing resources.
On gfx1101, the GPU conversion probe checks every half encoding across RGB output
channels and every unsigned R11/R10 input encoding, including padded input rows and
partial workgroups. Finite values and infinities match the original blits exactly;
NaN payloads may differ, but remain NaNs. Run the opt-in GPU regression with
`D4R_RUN_GPU_TESTS=1 python3 -m unittest discover -s tests -p test_output_conversion_gpu.py`.
Value 2 is diagnostic: output alternates every 300 frames, input every two such
blocks. Together they cycle through transfer/transfer, transfer/compute,
compute/transfer and compute/compute in one launch. Profile fields `vk_input_mode`
and `vk_output_mode` identify the actual paths of the older Vulkan timestamp slot,
not the current frame's mode. Set both flags to 1 for normal use after comparison.
`scripts/summarize_d4r_profile.py LOG --since-clock SECONDS --until-clock SECONDS`
can isolate a verified gameplay interval. Do not include startup, descriptor warmup,
concurrent GPU tests, menus or cutscenes in a performance comparison.