wouter/d4r
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

rr: add FSR Ray Regeneration, exact RDNA3 inference and motion experiments

Browse Source 
Add an experimental FSR Ray Regeneration backend for D3D12 RR requests:
- Convert depth, motion, normals and albedos into AMD denoiser inputs.
- Split and recompose diffuse/specular signals, then run Super Sampling.
- Add stability/detail controls, demodulation guards and halo limiting.
- Handle AMD provider discovery under NVIDIA GPU spoofing.
- Add runtime probes, synthetic reference scenes and per-dispatch profiling.

Add exact gfx11 DLSS-RR inference using f16 cooperative matrices:
- Reproduce FP8 conversions, accumulation and camera reprojection arithmetic.
- Widen model weights and avoid requiring VK_EXT_shader_float8.
- Extend model building, replay tools and descriptor metadata.

Expand motion preprocessing and experimental AW2 native-motion support:
- Add tiled LDS dilation, padded allocation handling and thin-only sky fill.
- Order dilation with input waits and NGX on the selected stream.
- Add jitter-aligned native composition, hybrid routing and wire history repair.
- Add prepared masks, sparse repair and independent motion/repair controls.

Add optional fused R11G11B10 input/output conversion, sampled stage timings
and profile summarization. Extend configuration, packaging, documentation,
FidelityFX headers and CPU/GPU regression coverage.
This commit is contained in:
ayo committed 2026-10-16 04:55:25 -04:00
1 parent 17c1828f97
commit 34e6b0fe71
83 files changed
+11300 -195
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

NOTICE
+9
View File
@@ -17,6 +17,15 @@ patches/vkd3d-proton/*.patch
under the GNU Lesser General Public License, version 2.1. These patches are offered under the
same license.
third_party/fidelityfx/
AMD FidelityFX SDK 2.3.0 API headers (ffx_api.h, ffx_api_types.h, ffx_denoiser.h), copied
unmodified; the SDK license lists them under the MIT license (third_party/fidelityfx/LICENSE.txt).
AMD's runtime DLLs are not included and remain under AMD's binary terms.
kernels/support/fsr_rr_prepare.comp
Its diffuse/specular split follows the DLSS-RR to FSR-RR adapter in Zach Hembree's OptiScaler
branch ffx-denoise-experimental and burak113's continuation (GPL-3.0).
Release zips (scripts/package_release.sh) also bundle binaries of ZLUDA (Apache-2.0 or MIT, with
LLVM under Apache-2.0 with LLVM exceptions), the patched vkd3d-proton (LGPL-2.1) and OptiScaler
(https://github.com/optiscaler/OptiScaler, GPL-3.0, unmodified). Each zip carries their license
README.md
+1
View File
@@ -62,6 +62,7 @@ PreferAccuracy = true
NativeSwinEncoders = false
MotionVectorDilation = 2
MotionDilationThinSky = false ; opt-in thin-only sky fill; keep false for current behavior
```
## Known issues
config/d4r.ini.default
+26 -1
View File
@@ -44,6 +44,24 @@ Enable = auto
; auto preserves the game's choice; default = NVIDIA's default, D/E = transformer models.
; These presets are independent of [DLSS] Model. E requires a DLL that contains that model.
Model = auto
; Which denoiser answers a game's Ray Reconstruction request (D4R_RR_BACKEND):
; dlss = NVIDIA's DLSS Ray Reconstruction (Model above), the default.
; fsr = experimental: AMD's FSR Ray Regeneration 1.2 (amd_fidelityfx_denoiser_dx12.dll, unmodified)
; denoises at render resolution, then Super Sampling ([DLSS] Model) upscales to the game's
; output size. D3D12 only. The game's combined colour is split into diffuse and specular
; by its albedos, which is an approximation; see docs/fsr-ray-regeneration.md.
; Changing it needs a game restart (or the game recreating its Ray Reconstruction feature).
Backend = dlss
; The AMD denoiser DLL for Backend = fsr (D4R_FSR_RR_DLL). auto: amd_fidelityfx_denoiser_dx12.dll in
; the d4r folder.
FsrDenoiser = auto
; Backend = fsr tuning. FsrStabilityBias: the AMD denoiser's temporal stability bias; auto = AMD's
; default (1), which blurs reflections and fine detail; 0 measured sharper and closer to the noise-free
; truth, moving or still (D4R_FSR_RR_STABILITY_BIAS). FsrDetail: the share (0-1) of the colour that goes
; around the denoiser, keeping detail it removes as noise (hair, water ripples, foliage) at the cost of some
; grain; 0 = off (D4R_FSR_RR_DETAIL).
FsrStabilityBias = auto
FsrDetail = 0
[Latency]
; How many frames old the upscaled image the game presents may be.
@@ -90,6 +108,10 @@ ImplicitMaxBlock = 256
; and same-frame linear VRAM inputs; unsupported features retain original vectors.
; Default off. May affect disocclusion edges. (D4R_MOTION_DILATION)
MotionVectorDilation = 0
; Opt-in sky-fill experiment for equal render/motion grids: borrow only from foreground
; runs <=3 pixels along either axis. Opaque dilation is unchanged. Restart to change.
; Unsupported routes retain nearest-foreground fill. (D4R_MOTION_THIN_SKY)
MotionDilationThinSky = false
; Keep DLSS inputs and output in VRAM (vkd3d-proton Vulkan interop) instead of copying through
; system memory. Falls back automatically when the game's resources do not qualify.
VramInterop = true
@@ -177,8 +199,11 @@ MangoHud = false
MangoHudConfig = fps,frametime,frame_timing=0,gpu_stats=0,cpu_stats=0,position=top-right,font_size=36
[Debug]
; Per-frame timing lines in the shim log (d4r_nvngx.log)
; Measure stage timings and write sampled timing lines in the shim log (d4r_nvngx.log)
Profile = false
; Log the first 5 frames, then every Nth frame. 1 = every frame. Restart after changing.
; FSR RR stage summaries average N completed GPU samples. Env: D4R_PROFILE_LOG_EVERY
ProfileLogEvery = 60
; Log directory (auto = ~/.cache/d4r-dlss-captures/game-<exe>-<time>)
LogDir = auto
docs/architecture.md
+1 -1
View File
@@ -112,7 +112,7 @@ OptiScaler 0.9.4 checks a relative `NvngxPath` against the process working direc
## Logs
The launcher writes each run's logs to `~/.cache/d4r-dlss-captures/game-<exe>-<time>/`. `d4r_nvngx.log` holds the shim's log. With `[Debug] Profile = true` it also records one `D4R_PROFILE` line per frame, with the stage timings.
The launcher writes each run's logs to `~/.cache/d4r-dlss-captures/game-<exe>-<time>/`. `d4r_nvngx.log` holds the shim's log. With `[Debug] Profile = true` it measures stage timings every frame and records sampled `D4R_PROFILE` lines. `ProfileLogEvery = 60` (the default) logs the first five frames and then every 60th frame; set it to `1` to log every frame, or another positive integer for a different interval. Restart the game after changing it. `D4R_PROFILE_LOG_EVERY` overrides the INI setting. FSR RR stage summaries average that many completed GPU samples; native Vulkan profiling uses the same log interval. Sampling reduces log writes while retaining profiling measurements and per-frame throughput intervals. `D4R_PROFILE_LOG=0` suppresses timing rows and FSR RR summaries.
Output completion uses event queries with 200 us NtDelayExecution sleeps by default. `D4R_SHIM_BLOCKING_SYNC = 0`
under `[Env]` restores context synchronization for an A/B check. The profile records
docs/fsr-ray-regeneration.md
+103
View File
@@ -0,0 +1,103 @@
# FSR Ray Regeneration as the Ray Reconstruction backend (experimental)
`[RayReconstruction] Backend = fsr` answers a game's DLSS Ray Reconstruction request (NGX feature 13)
with AMD's FSR Ray Regeneration 1.2 instead of NVIDIA's denoiser. `Backend = dlss` (the default) keeps
d4r's existing DLSS Ray Reconstruction. Changing the backend needs a game restart, or the game
recreating its Ray Reconstruction feature.
Status: works on an RX 7700 XT (gfx1101) under Proton with the synthetic test scene described below.
Alan Wake 2 logs from 2026-10-14 confirm denoiser dispatch and internal DLSS K upscaling, but the
game test reports excessive noise and artifacts. **Game image quality is not validated.**
AMD supports FSR Ray Regeneration only on RX 9000 / Windows 11. This
route is Linux/Proton only, and nothing here is validated on native Windows or for native Vulkan games.
## What runs
In the game's own command list and frame:
1. `kernels/support/fsr_rr_prepare.comp` (Vulkan, through vkd3d-proton's interop) turns the DLSS-RR
inputs into the denoiser's inputs:
- signed linear depth (from hardware or linear depth and the game's ViewToClip);
- previous-minus-current UV motion with the camera's view-z delta;
- octahedral normals plus roughness;
- linear albedos.
It splits the noisy colour into a diffuse and a specular signal by albedo and demodulates both
(after Zach Hembree's and burak113's GPL OptiScaler adapter). What cannot be demodulated, the sky,
and particles (when `DLSSD.ColorBeforeParticles` is given) go to a residual that bypasses the
denoiser.
2. AMD's `amd_fidelityfx_denoiser_dx12.dll`, unmodified, runs through `ffxDispatch`. Each lobe uses
the indirect path only when the game supplies that lobe's hit distance; otherwise it uses the
direct path.
3. `fsr_rr_compose.comp` remodulates the denoised signals and adds the residual.
4. An internal Super Sampling feature, configured like the user's own (`[DLSS] Model`, engine or
CUDA), upscales to the output size the game asked for.
`nvngx_dlssd.dll` is not used. When a frame cannot be denoised (missing guides or matrices, jittered
motion vectors, `DLSSD.OutputAlpha` requested), the evaluation fails and the reason is logged. It is
never upscaled noisy as if it had been denoised.
## Requirements
- AMD's `amd_fidelityfx_denoiser_dx12.dll` from FidelityFX SDK 2.3. Tested with SHA-256
`48f1e5888ba6a0a3d59a98b9751e37c392b0f7b8c223d0082d5c1f40642879d3`, file version 1.2.0.2740.
Put it in the d4r folder, or set `[RayReconstruction] FsrDenoiser`. It is AMD's binary under
AMD's terms, not part of this repository.
- d4r's vkd3d-proton built with `D4R_VKD3D_FSR_RR=1 scripts/build_vkd3d_proton_d4r.sh OUT`. This build
adds upstream dxil-spirv's accepted scalar FP8 fallback (PR 307) and extended emulation.
- `DXIL_SPIRV_CONFIG=wmma_rdna3_workaround` in the game's environment.
- Proton's `amdxc64.dll`. vkd3d-proton loads it only when the GPU is reported as AMD, so under
`PROTON_FORCE_NVAPI` the shim loads it after confirming through sysfs that the GPU is AMD's.
Without it, AMD's runtime offers no provider.
## Tuning
Two `[RayReconstruction]` keys adjust the denoiser. Both are off by default.
- `FsrStabilityBias` (`D4R_FSR_RR_STABILITY_BIAS`): AMD's temporal stability bias. `auto` keeps AMD's
default of 1. With that default, reflections and fine detail come out blurred. On Alan Wake 2's water,
the result had about half the detail of DLSS Ray Reconstruction. `0` brought the replayed water capture
from 29.5 to 36.6 dB PSNR against DLSS RR's output. On the synthetic scene it roughly halved the error
against ground truth, whether the camera was still, moving or just after a cut. GPU time was unchanged.
- `FsrDetail` (`D4R_FSR_RR_DETAIL`, 0–1): the share of the colour that skips the denoiser and goes
through the residual. It keeps detail the denoiser removes as noise: hair, water ripples and foliage.
The cost is some grain. With `0.25` on top of stability bias 0, PSNR against DLSS Ray Reconstruction's
output went up by 0.4 dB on the water capture, 1.4 dB on the street capture and 0.8 dB on a walking
capture in town. Frame-to-frame flicker off the water went from 0.0069 to 0.0073 (DLSS RR: 0.0060).
Two guards are on by default, because a game's single colour has light in it that the albedos did not
shape, such as fog, sky light, or the background behind a thin object's edge:
- No albedo below 0.1 divides a signal (`D4R_FSR_RR_DEMOD_FLOOR`). Dividing fog by dark foliage
overflowed, and the fog lost its colour and haze.
- Each denoised lobe is capped at the brightest noisy input of that lobe in its 3×3 neighbourhood
(`D4R_FSR_RR_HALO_LIMIT`, 0 turns it off). Without the cap, the denoiser leaks a thin object's large
demodulated signal across its edge, which shows as bright or blue fringes around poles, wires, hair
and the edges of water.
`D4R_FSR_RR_KEY_<n>=<value>` overrides any of AMD's six configuration keys, for diagnostics. The log
lists AMD's defaults when the denoiser context is created.
## Known limitations
- One noisy colour cannot be split exactly into diffuse and specular. On the test scene this route
flattens some shading gradients compared with DLSS Ray Reconstruction.
- Normals are assumed to be in world space. The depth delta comes from the camera's motion only, so
moving objects get no depth-change term.
- Guide subrect origins, jittered motion vectors and alpha output are not supported.
- Cost scales with render resolution. AMD's denoiser alone, on gfx1101:
- about 4 ms at 960×540 (2 signals);
- about 8 ms at 1280×720;
- about 26 ms at 1920×1080 (4 signals).
## Checks
- `tools/fsr_rr_probe.cpp` (`scripts/run_fsr_rr_probe_proton.sh`) tests AMD's runtime alone: provider,
context, multi-frame denoising measured against ground truth, and reset.
- `tools/rr_backend_check.cpp` (`scripts/run_rr_backend_check_proton.sh`) drives a feature-13 request
through the shim and compares the output with the scene rendered noise-free at output resolution.
`--feature 1` gives the Super-Sampling-only baseline. With `D4R_PROFILE=1`, the log reports GPU time
per stage.
- `--hw-depth --separate-eval-params` checks callers that put depth/roughness modes only on the
creation parameter object. The FSR adapter uses the resolved creation/per-frame settings; it
must not silently interpret hardware depth as linear when a frame omits the creation setting.
docs/native-kernels.md
+172 -2
View File
@@ -74,9 +74,19 @@ Initial validation: a gfx1101 fast L build completed two Ultra Performance harne
**Ray Reconstruction, the denoiser (`kernels/tex` + `cuda_dldn_engine_*`).**
- **Network:** `nvngx_dlssd.dll` defines the DLSS-D denoiser: six Swin encoders (`cuda_dldn_engine_swin_enc0_kernel` … `enc5`), five decoders (`dec0` … `dec4`), a high-fidelity kernel-prediction network whose output kernel is `cuda_dldn_engine_hkpn_output_kernel_transformer`, and the luma/auto-exposure/`reduce_sum` helpers. It shares no kernel with the Super Resolution libraries.
- **Native today:** the hkpn output kernel (and its `_diamond_wallaby` twin), built by `kernels/build.sh rr` as a wave64 surface-store replacement. `make_ptx.py` replaces `sust.b.2d.v4.b16.zero` — a whole RGBA16F texel — with `d4r_sust_v4b16`; the neural body and the texture reads are unchanged.
- **Still translated:** all eleven Swin layers. ZLUDA compiles NVIDIA's PTX at load time; the table below measures the resulting kernels' GPU execution, not JIT latency. A native replacement must preserve each layer's attention, norm and merge stages, as `kernels/m/rrlite_*` does for the upscaler.
- **CUDA route:** all eleven Swin layers remain translated. ZLUDA compiles NVIDIA's PTX at load time; the table below measures the resulting kernels' GPU execution, not JIT latency. A native replacement must preserve each layer's attention, norm and merge stages, as `kernels/m/rrlite_*` does for the upscaler.
- **Manifest:** the denoiser DLL is passed to `kernel_manifest.py` as `--rr-dll`, because its hashes may only authorize `cuda_dldn_engine_*` replacements. Without that split, a name both libraries define — `dl4rt_input_kernel` — would let a super-resolution binary be served for the denoiser's different implementation of it.
The separate [Vulkan RR engine](../engine/rr/README.md) also has an exact RDNA3 build: `D4R_RR_GFX11=1
D4R_RR_RELAXED=0 engine/rr/build_native_model.sh ...`. It runs the eleven native network stages with f16 cooperative
matrices and reproduces ZLUDA's FP8 lowering, without `VK_EXT_shader_float8`. On an RX 7700 XT, preset D, AW2 town
replay at 853×480 → 2560×1440, two engine runs and the CUDA reference are byte-identical over all 24 RGBA16F frames.
Replay GPU time is about 11.4 ms versus 27.9 ms for the current CUDA control; these are replay measurements.
Live AW2 denoiser GPU time was about 27.3 → 12.0 ms; whole-frame intervals were 42.6 → 24.4 ms in the faster
sessions and 53.4 → 36.2 ms in the slower sessions (one launch per backend in each state). The slower game state
remains; these figures are not pooled across states. The model's weights stay local. Select `[RayReconstruction] Backend = dlss` and `[Engine] Enabled = true` with the
model in `d4r/engine/rr-d`; FSR RR retains its own route. Preset E and other RDNA3 GPUs remain unverified.
Measured on the RX 9070 XT (gfx1201, RDNA4), Ray Reconstruction preset E,
1280×720 → 3840×2160, with `D4R_CUDA_KERNEL_PROFILE` in the D3D12 harness:
The earlier 20-frame run excluded the first 6 frames, with native hkpn output
@@ -295,6 +305,166 @@ Runs of the same build vary by about ±3% per kernel. M additionally skips its o
`[Interop] MotionVectorDilation = 1` or `2` (`D4R_MOTION_DILATION`) extends nearby foreground motion into pixels whose depth otherwise belongs to the background. The value selects the opaque-surface radius in render pixels. Both enabled values also fill sky gaps using the nearest foreground pixel within an eight-render-pixel circle (far depth 0 for reversed Z, 1 for normal Z). This is the wider sky fill from the comparison; it can strengthen trails in some frames and remains opt-in through `MotionVectorDilation`. This is a depth-guided approximation to better thin-feature motion coverage, not engine-side conservative rasterization. It improved the captured Townfall rooftop where the earlier native-encoder fallback did not resolve the artifact.
The d4r-owned GPU kernel preserves the original depth/motion buffers and writes a separate RG16F velocity texture. The worker waits for its completion before NGX evaluates. No per-frame CPU readback is needed. It requires same-frame linear VRAM inputs, full-resolution unjittered motion vectors and zero subrect bases. Unsupported features or unavailable kernels keep the original vectors. The default is `0` (off); larger radii can affect disocclusion and unrelated foreground edges. The method does not recreate geometry missing from every input frame or establish RTX parity.
The d4r-owned GPU kernel preserves the original depth/motion buffers and writes a separate RG16F velocity texture. The input marker precedes dilation, and dilation precedes NGX on the same bridge-selected stream (including the optional priority stream); there is no dilation-specific CPU completion wait. No per-frame CPU readback is needed. It requires same-frame linear VRAM inputs, render- or display-resolution unjittered motion vectors and zero subrect bases. Unsupported features or unavailable kernels keep the original vectors. The default is `0` (off); larger radii can affect disocclusion and unrelated foreground edges. The method does not recreate geometry missing from every input frame or establish RTX parity.
`scripts/build_wine_nvcuda_bridge.sh` embeds the support kernels when `D4R_ROCM_DIR` points to ROCm clang. All six gfx110x/gfx120x targets compile; physical validation is currently gfx1101 only. A bridge built without the support kernels reports the feature unavailable rather than changing inputs. Optional GPU regressions cover forward/reversed depth, two radii, jitter, padded rows and preservation of input buffers; see `tests/test_motion_kernel_gpu.py`.
For equal motion/render grids, zero jitter and radius 1 or 2, the bridge selects
`d4r_motion_dilate_lds`: 16x16 output tiles use a 32x32 depth halo and wave32 row
foreground masks. The nearest sky fill preserves the generic kernel's exact
(distance squared, x, y) tie-break. The additional padded export keeps the original
72-byte parameter structure and passes allocation height separately; inactive
rows/columns, including row-pitch padding, copy through from the original motion
buffer in the same launch. Display-resolution or jitter-mapped grids retain
`d4r_motion_dilate` and its existing padding-copy fallback.
Deferred CPU marker polling remains enabled with dilation. GPU input waits can
also run with the LDS path once that slot has successfully allocated its output
texture and submitted a dilation through the new bridge. Cold slots, changed
allocation geometry, older bridges and the generic path use CPU marker polling
so lazy allocation/module loading or synchronous fallback copies cannot block
behind an unreleased GPU wait. Existing RR auxiliary-upload and staged-input
synchronizations remain in place.
`motion_dilation_wall` measures host preparation/submission time after this
change, including allocation on cold slots; it no longer waits for the kernel to
finish. Compare pipeline/throughput fields for end-to-end cost. On the GPU-wait
route, the evaluation event precedes the input wait and dilation, so
`dlss_gpu` includes dilation after subtracting input readiness; on the CPU-polled
route its start event follows dilation. Those GPU timing scopes are different.
GPU regression tests compare both kernels byte-for-byte on deterministic random
inputs and independently check image/block-edge sky ties, both depth directions,
both radii, padded rows/bottom rows, guards and immutable inputs.
### Opt-in thin-only sky fill
`[Interop] MotionDilationThinSky = true` (`D4R_MOTION_THIN_SKY=1`) selects
`d4r_motion_dilate_lds_thin` when dilation is enabled and the equal-grid, zero-jitter
LDS route is available. The default is false and the existing nearest-foreground
behavior remains unchanged. Opaque radius-1/radius-2 dilation is unchanged.
The sky search accepts foreground runs of at most three pixels along **either**
axis. This retains long horizontal wires and vertical poles; requiring short runs
in both axes would discard them. A three-pixel mask halo beyond the search tile
prevents workgroup boundaries from truncating run classification. Nearest-distance
and coordinate tie-breaking remain the same among eligible candidates. Mapped-grid
or older-bridge routes retain nearest fill and report that thin fill is unsupported.
Both sky-fill modes currently assume normalized depth with sky at 0 (reversed) or
1 (forward). The available Alan Wake 2 RR captures have forward depth greater than
1 throughout; they contain no foreground according to this existing threshold and
cannot validate a sky-fill quality benefit. Thin mode is an experimental hypothesis,
not a measured in-game quality improvement. It stays opt-in; no automatic render-ratio
policy has been enabled. Restart after changing this setting.
### Experimental AW2 native motion coverage
The private AW2 driver prototype can feed output-resolution raster motion to
NGX with `D4R_NATIVE_MV_LIVE=1`. Its live path also uses a thin-silhouette halo:
`D4R_NATIVE_MV_THIN_HALO=0` selects the original 3x3 native compositor instead.
The halo extends only sky pixels toward foreground runs shorter than five output
pixels on at least one axis. Its radius tracks two render pixels, capped at six
output pixels. Thick surface interiors do not receive that extra expansion.
The original compositor and 80-byte parameter ABI remain available; the Wine
bridge exports the additional `d4rComposeNativeMotionThin` operation.
This requires the private native-raster driver and GPU-ordered owned-input
validity contract. It is experimental, defaults remain unchanged, and this is not
a general engine-independent native-velocity feature. A 24-frame AW2 forward-walk
experiment at 853x480 -> 2560x1440 reduces paired wire dropout by 67.5%, beading by
22.7%, temporal contrast error by 23.2%, and nearby darkening proxy by 39.9% after
eight warm-up frames. These are one-scene proxies, without native-color ground
truth, and do not establish artifact-free rendering. On gfx1101 the complete
halo compositor takes 1.175 ms kernel-only versus 0.148 ms for the original
native compositor; no matched live performance gain is claimed.
The private AW2 path also supports `D4R_NATIVE_MV_HYBRID=1`: retain native
motion within six output pixels of a thin native silhouette and preserve the
game's nearest-sampled motion elsewhere. This avoids substituting the complete
native geometry/camera field over distant surfaces. It requires the additive
`d4rComposeNativeMotionHybrid` bridge operation and matching low-resolution motion
dimensions; unsupported layouts retain the full native fallback. It defaults off.
`D4R_NATIVE_MV_WIRE_SKY_ONLY=1` restricts color repair to silhouettes with sky on
both sides of the thin axis, also default off. Fractional coverage also checks the
nearest native depth: a different foreground surface or NaN depth is preserved,
rather than receiving color from an adjacent wire. Depth-matched wire centers and
sky remain eligible. These are experimental controls,
not an artifact-free rendering or 2.47 ms performance claim.
Wire-color repair is independent of the extra motion halo. With a matching
bridge, `D4R_NATIVE_MV_THIN_HALO=0` selects jitter-aligned 3x3 native composition
while leaving `D4R_NATIVE_MV_WIRE_HISTORY` active. Earlier builds gated both
operations on the halo flag, which prevented an isolated halo-off comparison.
All these environment controls belong in the per-game INI's `[Env]` section.
`D4R_NATIVE_MV_SUBSTITUTE=0` decouples wire repair from NGX motion substitution.
With `D4R_NATIVE_MV_LIVE=1` and `D4R_NATIVE_MV_WIRE_HISTORY=1`, the native field
is composed only for repair; NGX retains the game's vectors, creation flags and
MV scale. Dilation still follows its existing setting; use
`D4R_MOTION_DILATION=0` for an unmodified game-vector comparison. `SUBSTITUTE`
defaults to 1, preserving the combined path. `LIVE=0` still bypasses both native
composition and repair. This is an experimental isolation control: repair-only
can still alter distant thin silhouettes and has not established visual stability.
Keep CPU input waiting for these comparisons; no GPU-wait safety gate is removed.
Profile lines also identify the feature, native-field readiness, actual native
substitution, requested repair and requested GPU completion-word path. Use
`scripts/summarize_d4r_profile.py LOG` to group warm samples by feature instance,
resolution, model, quality, flags and routing. Missing GPU timings remain missing;
CPU marker/pipeline times and delayed Vulkan samples are summarized independently.
With profiling enabled, sampled live-native frames also report
`native_motion_gpu` and `wire_repair_gpu`. These bracket the composition kernel
and color-repair dispatches on the NGX stream; elapsed times are read after the
existing publication synchronization. They add no CPU readiness wait and remain
`-1` when inactive or unavailable. They describe the current frame, while the
existing game-queue Vulkan timestamps are read from older completed slots.
`D4R_NATIVE_MV_PREPARED=1` selects an exact implementation of the existing full
native-motion and wire-repair policy. A 32x32-tile preparation kernel builds a
packed native silhouette mask and per-pixel wire axes once; a wave64 kernel
composes the same motion field. Sparse repair consumes those axes instead of
repeating the depth classification. RGB/depth history entries marked age 255
are left unspecified because the consumer checks age before reading them;
rendered RGBA and readable history remain unchanged. The additive bridge
operations are `d4rComposeNativeMotionPrepared` and
`d4rStabilizeThinWiresPrepared`; missing exports, incompatible flags/layouts or
allocation failure retain the earlier implementation. The generic default is
0; enable it under `[Env]` and restart for an A/B test. This is a performance
implementation switch and does not enable hybrid motion or sky-only repair.
On gfx1101, matched replay tests measured roughly 0.28 ms for native motion plus
shared preparation and 0.17 ms for sparse repair/apply, excluding NGX. These are
kernel replay measurements, not a live total or a claim that distant artifacts
are resolved. The sampled `native_motion_gpu` field includes the shared wire
preparation when this route is active.
The prepared compositor also remains available with `WIRE_HISTORY=0`; turning
color repair off does not require the slower legacy native compositor.
### Optional R11G11B10 interop conversion
`D4R_SHIM_FUSE_R11_INPUT=1` samples an R11G11B10 color image directly into the
pitched RGBA16F input buffer. `D4R_SHIM_FUSE_R11_OUTPUT=1` writes the RGBA16F result
buffer directly into an R11G11B10 storage image. Both default off. These reuse the
Vulkan staging shaders to remove intermediate-image copies and blits; NGX model,
input readiness, frame age and temporal state are unchanged. Unsupported resource
usage or descriptor allocation failure retains the original transfer path. Output
requires `ALLOW_UNORDERED_ACCESS`; input requires shader-resource access.
Descriptors and owner images remain alive through recorded allocator retirement,
with bounded retention for changing resources.
On gfx1101, the GPU conversion probe checks every half encoding across RGB output
channels and every unsigned R11/R10 input encoding, including padded input rows and
partial workgroups. Finite values and infinities match the original blits exactly;
NaN payloads may differ, but remain NaNs. Run the opt-in GPU regression with
`D4R_RUN_GPU_TESTS=1 python3 -m unittest discover -s tests -p test_output_conversion_gpu.py`.
Value 2 is diagnostic: output alternates every 300 frames, input every two such
blocks. Together they cycle through transfer/transfer, transfer/compute,
compute/transfer and compute/compute in one launch. Profile fields `vk_input_mode`
and `vk_output_mode` identify the actual paths of the older Vulkan timestamp slot,
not the current frame's mode. Set both flags to 1 for normal use after comparison.
`scripts/summarize_d4r_profile.py LOG --since-clock SECONDS --until-clock SECONDS`
can isolate a verified gameplay interval. Do not include startup, descriptor warmup,
concurrent GPU tests, menus or cutscenes in a performance comparison.
engine/compile_rr.py
+5 -1
View File
@@ -56,8 +56,12 @@ NOOPT = {'enc1'}
SURFACE_FORMATS = {'dec0': {144: 'rgba8_snorm'}}
# D4R_RR_GFX11=1: shaders for RDNA3, which has no FP8 matrices or VK_EXT_shader_float8 (engine/rr/rr_gfx11.glsl)
GFX11 = ['-DRR_GFX11'] if os.environ.get('D4R_RR_GFX11') == '1' else []
def glsl(source, output, *defines):
subprocess.run(['glslangValidator', '-V', '--target-env', 'vulkan1.3', *defines, str(source), '-o', str(output)],
subprocess.run(['glslangValidator', '-V', '--target-env', 'vulkan1.3', *GFX11, *defines, str(source), '-o', str(output)],
check=True, stdout=subprocess.DEVNULL)
engine/evk.cpp
+12 -2
View File
@@ -54,8 +54,18 @@ int main(int argc, char** argv)
uint32_t nq = 8; VkQueueFamilyProperties qf[8]; vkGetPhysicalDeviceQueueFamilyProperties(pd, &nq, qf);
uint32_t qi = 0; for (uint32_t i = 0; i < nq; ++i) if (qf[i].queueFlags & VK_QUEUE_COMPUTE_BIT) { qi = i; break; }
float prio = 1; VkDeviceQueueCreateInfo qci{VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO}; qci.queueFamilyIndex = qi; qci.queueCount = 1; qci.pQueuePriorities = &prio;
const char* exts[] = {"VK_KHR_cooperative_matrix", "VK_EXT_shader_float8"};
VkDeviceCreateInfo dci{VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO, &f2}; dci.queueCreateInfoCount = 1; dci.pQueueCreateInfos = &qci; dci.enabledExtensionCount = 2; dci.ppEnabledExtensionNames = exts;
uint32_t ne = 0; CK(vkEnumerateDeviceExtensionProperties(pd, nullptr, &ne, nullptr));
std::vector<VkExtensionProperties> ep(ne); CK(vkEnumerateDeviceExtensionProperties(pd, nullptr, &ne, ep.data()));
const bool hasFloat8 = std::any_of(ep.begin(), ep.end(), [](const auto& e) {
return strcmp(e.extensionName, VK_EXT_SHADER_FLOAT8_EXTENSION_NAME) == 0;
});
if (!hasFloat8) f8.shaderFloat8 = f8.shaderFloat8CooperativeMatrix = VK_FALSE;
std::vector<const char*> exts;
if (cm.cooperativeMatrix) exts.push_back(VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME);
if (f8.shaderFloat8) exts.push_back(VK_EXT_SHADER_FLOAT8_EXTENSION_NAME);
if (!cm.cooperativeMatrix) v13.pNext = nullptr; else if (!f8.shaderFloat8) cm.pNext = nullptr;
VkDeviceCreateInfo dci{VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO, &f2}; dci.queueCreateInfoCount = 1; dci.pQueueCreateInfos = &qci;
dci.enabledExtensionCount = static_cast<uint32_t>(exts.size()); dci.ppEnabledExtensionNames = exts.data();
VkDevice dev; CK(vkCreateDevice(pd, &dci, nullptr, &dev));
VkQueue q; vkGetDeviceQueue(dev, qi, 0, &q);
std::vector<VkDescriptorSetLayoutBinding> lb;
engine/replay_rr.cpp
+21 -4
View File
@@ -9,6 +9,7 @@
// (RGBA8 UNORM), all at the render size.
// OUT_DIR receives F-NAME.bin for every debug image of frame F.
#include "runtime.h"
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <fstream>
@@ -124,8 +125,16 @@ int main(int argc, char** argv) try {
VkPhysicalDeviceVulkan11Features v11{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VULKAN_1_1_FEATURES, &v12};
VkPhysicalDeviceFeatures2 f2{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2, &v11};
vkGetPhysicalDeviceFeatures2(c.physical, &f2);
const char* extensions[] = {VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME, VK_EXT_SHADER_FLOAT8_EXTENSION_NAME,
VK_EXT_ROBUSTNESS_2_EXTENSION_NAME};
uint32_t ne = 0; CK(vkEnumerateDeviceExtensionProperties(c.physical, nullptr, &ne, nullptr));
std::vector<VkExtensionProperties> ep(ne); CK(vkEnumerateDeviceExtensionProperties(c.physical, nullptr, &ne, ep.data()));
auto hasExtension = [&](const char* name) {
return std::any_of(ep.begin(), ep.end(), [&](const auto& e) { return std::strcmp(e.extensionName, name) == 0; });
};
if (!hasExtension(VK_EXT_SHADER_FLOAT8_EXTENSION_NAME))
f8.shaderFloat8 = f8.shaderFloat8CooperativeMatrix = VK_FALSE;
std::vector<const char*> extensions;
if (cm.cooperativeMatrix) extensions.push_back(VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME);
if (f8.shaderFloat8) extensions.push_back(VK_EXT_SHADER_FLOAT8_EXTENSION_NAME);
uint32_t families = 8; VkQueueFamilyProperties qf[8]; vkGetPhysicalDeviceQueueFamilyProperties(c.physical, &families, qf);
uint32_t family = 0;
for (uint32_t i = 0; i < families; ++i) if (qf[i].queueFlags & VK_QUEUE_COMPUTE_BIT) { family = i; break; }
@@ -135,8 +144,16 @@ int main(int argc, char** argv) try {
// D4R_RR_TEST_ROBUST2=1: robustBufferAccess2 / robustImageAccess2 as vkd3d-proton enables them (out-of-bounds
// reads return zero)
const bool robust2 = std::getenv("D4R_RR_TEST_ROBUST2") != nullptr;
if (!robust2) { rb2.robustBufferAccess2 = rb2.robustImageAccess2 = rb2.nullDescriptor = VK_FALSE; }
di.enabledExtensionCount = robust2 ? 3 : 2; di.ppEnabledExtensionNames = extensions;
if (robust2 && hasExtension(VK_EXT_ROBUSTNESS_2_EXTENSION_NAME)) extensions.push_back(VK_EXT_ROBUSTNESS_2_EXTENSION_NAME);
else { rb2.robustBufferAccess2 = rb2.robustImageAccess2 = rb2.nullDescriptor = VK_FALSE; }
// Omit unsupported extension feature structures as well as their extension names. gfx11 RR uses f16 matrices.
v13.pNext = cm.cooperativeMatrix ? static_cast<void*>(&cm) : nullptr;
cm.pNext = f8.shaderFloat8 ? static_cast<void*>(&f8) : nullptr;
f8.pNext = robust2 && hasExtension(VK_EXT_ROBUSTNESS_2_EXTENSION_NAME) ? &rb2 : nullptr;
if (robust2 && hasExtension(VK_EXT_ROBUSTNESS_2_EXTENSION_NAME) && !f8.shaderFloat8) {
if (cm.cooperativeMatrix) cm.pNext = &rb2; else v13.pNext = &rb2;
}
di.enabledExtensionCount = static_cast<uint32_t>(extensions.size()); di.ppEnabledExtensionNames = extensions.data();
// D4R_RR_TEST_NO_ROBUST=1: no robust buffer or image access (out-of-bounds reads are undefined)
if (std::getenv("D4R_RR_TEST_NO_ROBUST")) { f2.features.robustBufferAccess = VK_FALSE; v13.robustImageAccess = VK_FALSE; }
// D4R_RR_TEST_NO_COOPMAT=1: a device like vkd3d-proton's, without the cooperative-matrix and float8 features
engine/rr/README.md
+31
View File
@@ -15,6 +15,37 @@ NVIDIA kernel of the frame; the engine records them in NVIDIA's order:
The D variants of enc0, dec0 and the output kernel are different code; the other nine kernels are shared and only
their weights differ between presets.
## RDNA3 exact preset D (2026-10-14)
Build with `D4R_RR_GFX11=1 D4R_RR_RELAXED=0 engine/rr/build_native_model.sh CAPTURE OUT DLL`.
`rr_gfx11.glsl` reproduces ZLUDA's gfx11 FP8 widening, K positions, scaled f32 accumulators and final binary16
rounding using f16 cooperative matrices. The weight files include a widened copy. Relaxed and mix arithmetic are
unavailable in this build; `VK_EXT_shader_float8` is unnecessary. `kvk`, `evk` and `d4r-rr` enable float8 only when
the driver advertises the extension, because RADV gfx11 may report its feature bits without the extension.
`arithmetic gfx11` in `rr.txt` also selects NGX 310.7's CPU camera operation order. Its inversions, relative camera
translation and intermediate reprojection products leave small residues that affect specular motion history;
double-precision equivalents were insufficient for temporal equality. Other model builds keep their existing path.
RX 7700 XT (gfx1101), RADV, preset D, AW2 town at 853×480 → 2560×1440: enc0 and hkpn are byte-identical on all three
captured launches, joining the ten previously verified stages. Encoder surface 112's undefined alpha is excluded
from the isolated stage check. The complete presented RGBA16F output, including alpha, is byte-identical over all
24 replay frames in two engine runs versus the deterministic CUDA control. Engine GPU time is about 11.4 ms versus
27.9 ms for the current control; warmed samples varied by about 0.1 ms within each engine run. Older CUDA runs
included timing outliers, so their lower 26.7 ms figure is historical, not the paired control. No frame-age or
resolution change contributes to the result. Other RDNA3 GPUs, preset E, and other games require validation.
AW2 live at the same setting: CUDA / engine denoiser GPU medians 27.3 / 12.0 ms; frame intervals 42.6 / 24.4 ms
in the faster sessions and 53.4 / 36.2 ms in the slower sessions. These are separate session states, one launch per
backend in each state; the engine does not resolve the pre-existing slower game state. Frame intervals use the
frame-count/time slope over the final 30 seconds, with sampled instantaneous spreads of about 1 ms for CUDA and
3-4 ms for the engine. The same-frame engine is opt-in; configuration was restored after testing.
Put the local model under `d4r/engine/rr-d`, or set `D4R_ENGINE_MODEL_RR_D_DIR` to its Windows path. The shim runs it
with `[RayReconstruction] Backend = dlss` and `[Engine] Enabled = true`. `D4R_ENGINE_RR_PROFILE` log lines give
game-queue GPU time around the complete engine evaluation; `gpu_frame` identifies the earlier frame whose query
has completed, while `game_interval` is the current call interval. Query readback does not delay the image.
## Stage contract
Every stage shader reads the same data NVIDIA's kernel reads, in the same layouts, so each one can be checked on
engine/rr/build_native_model.sh
+33 -6
View File
@@ -74,10 +74,23 @@ if (( D4R_RR_RELAXED > 0 )); then
fi
# D4R_RR_MIX=1: the 32-deep steps round once through v_fma_mix (the shaders' K32): ~5% faster, not provably bit-exact
[[ "${D4R_RR_MIX:-0}" == 1 ]] && DEFS+=(-DRR_MIX)
glsl() { # glsl OUT.spv SOURCE INCLUDE_DIR...
# D4R_RR_GFX11=1: RDNA3 (no FP8 matrices, no VK_EXT_shader_float8): the stages reproduce ZLUDA's gfx11 lowering bit for
# bit (engine/rr/rr_gfx11.glsl); exact arithmetic only, and each native weight file carries a widened copy (W_WIDE)
GFX11=0
if [[ "${D4R_RR_GFX11:-0}" == 1 ]]; then
GFX11=1; DEFS=(-DRR_EXACT_CONVERT -DRR_GFX11); [[ "$PRESET" == 4 ]] && DEFS+=(-DRR_PRESET_D)
(( D4R_RR_RELAXED == 0 )) && [[ "${D4R_RR_MIX:-0}" != 1 ]] ||
echo "D4R_RR_GFX11: relaxed / mix builds do not exist for RDNA3; building the exact model" >&2
D4R_RR_RELAXED=0
fi
# The gfx11 runtime also reproduces NGX's CPU camera arithmetic, whose residues feed temporal specular motion.
(( GFX11 == 0 )) || sed -i '/^preset /a arithmetic gfx11' "$OUT/rr.txt"
# widen WEIGHTS: on RDNA3, append the widened copy and print -DW_WIDE=...; nothing otherwise
widen() { (( GFX11 )) && printf -- '-DW_WIDE=%su' "$(python3 "$RR/widen_weights.py" "$1")" || true; }
glsl() { # glsl OUT.spv SOURCE INCLUDE_DIR... [-DEXTRA]
local out="$1" src="$2"; shift 2
local dir inc=(); for dir in "$@"; do inc+=("-I$dir"); done
"${CAPPED[@]}" glslangValidator --target-env vulkan1.3 -V "${DEFS[@]}" "${inc[@]}" -o "$out" "$src" > "$WORK/glsl.log" 2>&1 ||
local dir inc=() extra=(); for dir in "$@"; do [[ "$dir" == -D* ]] && extra+=("$dir") || inc+=("-I$dir"); done
"${CAPPED[@]}" glslangValidator --target-env vulkan1.3 -V "${DEFS[@]}" "${extra[@]}" "${inc[@]}" -o "$out" "$src" > "$WORK/glsl.log" 2>&1 ||
{ grep -E 'ERROR' "$WORK/glsl.log" | head -5; exit 1; }
# from level 2 on the conversions are left to the driver (as compile_rr2.py does for preset F)
(( D4R_RR_RELAXED >= 2 )) || python3 "$RR/spv_exact_convert.py" "$out" > /dev/null
@@ -94,7 +107,7 @@ for s in enc1 enc2 enc3 enc4 enc5 dec4 dec3 dec2 dec1; do
cp "$WORK/dll/$s.npz" "$d/weights.npz"
[[ -z "$CAP" ]] || python3 "$RR/swin_ref.py" "$CAP" "$s" "$FRAME" "$d/weights.npz" --blocks "$b" 0,0 | tail -2 | sed "s/^/$s: /"
python3 "$RR/swin_pack.py" "$s" "$d/weights.npz" "$OUT/weights-${s}m.bin" "$d/swin_stage.h" > /dev/null
glsl "$OUT/${s}_m.spv" "$RR/swin_m.comp" "$d" "$RR"
glsl "$OUT/${s}_m.spv" "$RR/swin_m.comp" "$d" "$RR" $(widen "$OUT/weights-${s}m.bin")
native "$s" "weights-${s}m.bin"
done
@@ -103,7 +116,7 @@ d="$WORK/dec0"; mkdir -p "$d"
cp "$WORK/dll/dec0.npz" "$d/weights.npz"
[[ -z "$CAP" ]] || python3 "$RR/dec0_ref.py" "$CAP" "$FRAME" "$d/weights.npz" --blocks 5,5 0,0 | tail -2 | sed 's/^/dec0: /'
python3 "$RR/dec0_pack.py" "$d/weights.npz" "$OUT/weights-dec0m.bin" "$d/dec0_m_w.h" > /dev/null
glsl "$OUT/dec0_m.spv" "$RR/dec0_m.comp" "$d" "$RR"
glsl "$OUT/dec0_m.spv" "$RR/dec0_m.comp" "$d" "$RR" $(widen "$OUT/weights-dec0m.bin")
native dec0 weights-dec0m.bin
# encoder 0: the translated feature gather + the native rest
@@ -119,7 +132,7 @@ else
cp "$d/enc0_m.bin" "$OUT/weights-enc0m.bin"
fi
python3 "$RR/gen_enc0_m.py" "$OUT/enc0.generated.comp" "$d/enc0_m.comp" > /dev/null
glsl "$OUT/enc0_m.spv" "$d/enc0_m.comp" "$d" "$RR"
glsl "$OUT/enc0_m.spv" "$d/enc0_m.comp" "$d" "$RR" $(widen "$OUT/weights-enc0m.bin")
native enc0 weights-enc0m.bin
# the translated output stage (preset D; preset E's when engine/rr/hkpn.comp is not used) has no subgroup operations
@@ -127,5 +140,19 @@ if grep -q '"shader": "generated from the DLL"' <(python3 -c "import json,sys; p
sed -i -E 's/^stage hkpn hkpn\.spv 32 /stage hkpn hkpn.spv 64 /' "$OUT/rr.txt"
fi
for s in enc0 enc1 enc2 enc3 enc4 enc5 dec4 dec3 dec2 dec1 dec0; do rm -f "$OUT/weights-$s.bin"; done
# test_layer.py must bind optional images as well as the ones present in a capture. Keep a sidecar for every used
# shader, including native stages, so an absent input never shifts the shader's descriptor bindings.
python3 - "$OUT" <<'PY'
import json, sys
from pathlib import Path
out = Path(sys.argv[1])
for line in (out / 'rr.txt').read_text().splitlines():
fields = line.split()
if not fields or fields[0] != 'stage':
continue
textures = [] if fields[5] == '-' else list(map(int, fields[5].split(',')))
surfaces = [] if fields[6] == '-' else [[int(s.split(':')[0]), s.split(':')[1]] for s in fields[6].split(',')]
(out / fields[2]).with_suffix('.json').write_text(json.dumps({'textures': textures, 'surfaces': surfaces}) + '\n')
PY
printf 'Native Ray Reconstruction model (preset %s) in %s\n' "$([[ "$PRESET" == 4 ]] && echo D || echo E)" "$OUT"
grep '^stage' "$OUT/rr.txt" | awk '{ printf " %s %s (%s-wide)\n", $2, $3, $4 }'
engine/rr/dec0_m.comp
+39 -13
View File
@@ -17,7 +17,9 @@
#extension GL_EXT_shader_explicit_arithmetic_types : require
#extension GL_EXT_shader_16bit_storage : require
#extension GL_EXT_shader_8bit_storage : require
#ifndef RR_GFX11
#extension GL_EXT_float_e4m3 : require
#endif
#extension GL_KHR_shader_subgroup_basic : require
#extension GL_KHR_shader_subgroup_shuffle : require
#extension GL_EXT_control_flow_attributes : require
@@ -27,7 +29,9 @@ layout(local_size_x = 128) in;
#include "rr_exact.glsl"
layout(set = 0, binding = 0, std430) readonly buffer ParamBlock { uint p[]; } param;
#ifndef RR_GFX11
layout(set = 0, binding = 1, std430) readonly buffer W8 { floate4m3_t w[]; } w8;
#endif
layout(set = 0, binding = 1, std430) readonly buffer W16 { float16_t w[]; } w16;
layout(set = 0, binding = 1, std430) readonly buffer W16U { uint16_t w[]; } w16u;
layout(set = 0, binding = 1, std430) readonly buffer W8U { uint8_t w[]; } w8u;
@@ -40,10 +44,15 @@ layout(set = 0, binding = 7, rgba8_snorm) uniform writeonly image2D S144;
#define ACC coopmat<float, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#define HAC coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#ifndef RR_GFX11
#define A8 coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
#define B8 coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseB>
#endif
#define RM gl_CooperativeMatrixLayoutRowMajor
#define CM gl_CooperativeMatrixLayoutColumnMajor
#ifdef RR_GFX11
#include "rr_gfx11_k32.glsl"
#else
#define LDA(dst, base, k, mt, kt) coopMatLoad((dst), w8.w, (base) + 256u * ((mt) * ((k) / 16u) + (kt)), 16u, RM)
#define LDB(dst, base, n, kt, nt) coopMatLoad((dst), w8.w, (base) + 256u * ((kt) * ((n) / 16u) + (nt)), 16u, RM)
#define RND(m) [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (m)[i_] = rnd16((m)[i_]);
@@ -65,6 +74,7 @@ layout(set = 0, binding = 7, rgba8_snorm) uniform writeonly image2D S144;
#define K32(acc, a0, b0, a1, b1) { (acc) = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), (acc))); RND(acc) }
#define K32H(acc, a0, b0, a1, b1) { ACC p_ = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), ACC(acc))); [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (acc)[i_] = float16_t(p_[i_]); }
#endif
#endif
const uint HEAD = W_WK1 - W_WK0, BRANCH = W_W1_1 - W_W1_0;
// shared memory, aliased by phase: expand and prologue [LOW | EX], then the attention [K | V^T]
@@ -124,16 +134,14 @@ void main()
[[unroll]] for (uint kt = 0u; kt < 4u; ++kt) LDA(we[mm][kt], W_WE, 64u, 2u * wv + mm, kt);
[[unroll]] for (uint n = 0u; n < 3u; ++n) {
B8 lb[4];
[[unroll]] for (uint kt = 0u; kt < 4u; ++kt) coopMatLoad(lb[kt], LOW, n * 256u + 4u * kt, 16u, CM);
[[unroll]] for (uint kt = 0u; kt < 4u; ++kt) LDS8B(lb[kt], LOW, n * 256u + 4u * kt, 16u);
[[unroll]] for (uint mm = 0u; mm < 2u; ++mm) {
const uint m = 2u * wv + mm;
RAC a;
[[unroll]] for (int i = 0; i < 8; ++i) a[i] = RV(w16.w[W_BE / 2u + 16u * m + 8u * hl + uint(i)]);
K32(a, we[mm][0], lb[0], we[mm][1], lb[1])
K32(a, we[mm][2], lb[2], we[mm][3], lb[3])
B8 q;
Q8(q, a)
coopMatStore(q, POOL, EX_AT + n * 512u + 4u * m, 32u, CM);
Q8STB(a, POOL, EX_AT + n * 512u + 4u * m, 32u)
}
}
}
@@ -149,8 +157,12 @@ void main()
[[unroll]] for (uint wd = 0u; wd < 8u; ++wd) {
const uint e = POOL[EX_AT + r * 32u + 8u * s + wd], sk = arena.w[sb + wd];
[[unroll]] for (uint b = 0u; b < 4u; ++b) {
#ifdef RR_GFX11
precise float16_t x = g11_e4h((e >> (8u * b)) & 0xffu) + g11_e4h((sk >> (8u * b)) & 0xffu);
#else
precise float16_t x = float16_t(uintBitsToFloate4m3EXT(uint8_t((e >> (8u * b)) & 0xffu)))
+ float16_t(uintBitsToFloate4m3EXT(uint8_t((sk >> (8u * b)) & 0xffu)));
#endif
v[4u * wd + b] = x;
}
}
@@ -169,7 +181,7 @@ void main()
precise float16_t se = sm + float16_t(0.0001220703125);
const float16_t rr = rsq16(se);
[[unroll]] for (uint wd = 0u; wd < 8u; ++wd) {
floate4m3_t c[4];
F8 c[4];
[[unroll]] for (uint b = 0u; b < 4u; ++b) {
const uint j = 4u * wd + b;
precise float16_t g = rr * w16.w[W_GAIN0 / 2u + inter(j)];
@@ -186,7 +198,11 @@ void main()
RAC R[2];
[[unroll]] for (uint m = 0u; m < 2u; ++m) {
HAC x;
#ifdef RR_GFX11
[[unroll]] for (int i = 0; i < 8; ++i) x[i] = XR[wv * 512u + 16u * m + col * 32u + 8u * hl + uint(i)];
#else
coopMatLoad(x, XR, wv * 512u + 16u * m, 32u, CM);
#endif
[[unroll]] for (int i = 0; i < 8; ++i) {
precise float16_t c = float16_t(x[i]) + w16.w[W_TABLE0 / 2u + 16u * m + 8u * hl + uint(i)];
R[m][i] = RV(c);
@@ -201,21 +217,17 @@ void main()
A8 wk[2], xa[2];
B8 wvb[2], xb[2];
[[unroll]] for (uint kt = 0u; kt < 2u; ++kt) {
coopMatLoad(xb[kt], XT, n * 128u + 4u * kt, 8u, CM);
coopMatLoad(xa[kt], XT, n * 128u + 4u * kt, 8u, RM);
LDS8B(xb[kt], XT, n * 128u + 4u * kt, 8u);
LDS8A(xa[kt], XT, n * 128u + 4u * kt, 8u);
LDA(wk[kt], W_WK0 + HEAD * hd, 32u, c, kt);
LDB(wvb[kt], W_WV0 + HEAD * hd, 32u, kt, c);
}
RAC a = RAC(RV(float16_t(0.0)));
K32(a, wk[0], xb[0], wk[1], xb[1])
A8 ka;
Q8(ka, a)
coopMatStore(ka, POOL, n * 128u + 4u * c, 8u, RM);
Q8STA(a, POOL, n * 128u + 4u * c, 8u)
RAC v = RAC(RV(float16_t(0.0)));
K32(v, xa[0], wvb[0], xa[1], wvb[1])
A8 va;
Q8(va, v)
coopMatStore(va, POOL, 768u + c * 384u + 4u * n, 24u, RM);
Q8STA(v, POOL, 768u + c * 384u + 4u * n, 24u)
}
barrier();
#include "rr_attn.glsl"
@@ -228,6 +240,9 @@ void main()
// ---------------------------------------------------------------- MLP
B8 Nb[2];
#ifdef RR_GFX11
F8V NbC[2] = F8V[2](uvec2(0u), uvec2(0u));
#endif
{
float16_t rv[2][8];
[[unroll]] for (uint m = 0u; m < 2u; ++m) [[unroll]] for (int i = 0; i < 8; ++i) rv[m][i] = float16_t(R[m][i]);
@@ -255,10 +270,17 @@ void main()
const uint k = 16u * m + 8u * hl + uint(i);
precise float16_t g = r * w16.w[W_GAIN1 / 2u + k];
precise float16_t n = rv[m][i] * g;
#ifdef RR_GFX11
NbC[m][i >> 2] |= q8h(n) << (8u * uint(i & 3));
#else
Nb[m][i] = q8h(n);
#endif
precise float16_t a = rv[m][i] + w16.w[W_TABLE1 / 2u + k];
R[m][i] = RV(a);
}
#ifdef RR_GFX11
[[unroll]] for (uint m = 0u; m < 2u; ++m) G11_B(Nb[m], NbC[m])
#endif
}
for (uint br = 0u; br < 4u; ++br) {
const uint w1 = W_W1_0 + BRANCH * br, b1 = W_B1_0 + BRANCH * br, w2 = W_W2_0 + BRANCH * br;
@@ -269,7 +291,11 @@ void main()
A8 a0, a1;
LDA(a0, w1, 32u, c, 0u); LDA(a1, w1, 32u, c, 1u);
K32(h, a0, Nb[0], a1, Nb[1])
#ifdef RR_GFX11
{ F8V hc_; ACTQ8C(hc_, h) G11_B(Hb[c], hc_) }
#else
ACTQ8(Hb[c], h)
#endif
}
[[unroll]] for (uint m = 0u; m < 2u; ++m) {
A8 a0, a1;
engine/rr/enc0_m_decl.glsl
+10
View File
@@ -7,17 +7,24 @@
#extension GL_GOOGLE_include_directive : require
#include "enc0_m_w.h"
#include "rr_exact.glsl"
#ifndef RR_GFX11
layout(set = 0, binding = 1, std430) readonly buffer W8 { floate4m3_t w[]; } w8;
#endif
layout(set = 0, binding = 1, std430) readonly buffer W16 { float16_t w[]; } w16;
layout(set = 0, binding = 1, std430) readonly buffer W16U { uint16_t w[]; } w16u;
layout(set = 0, binding = 1, std430) readonly buffer W8U { uint8_t w[]; } w8u;
layout(set = 0, binding = 2, std430) buffer Arena { uint w[]; } arena;
#define ACC coopmat<float, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#define HAC coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#ifndef RR_GFX11
#define A8 coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
#define B8 coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseB>
#endif
#define RM gl_CooperativeMatrixLayoutRowMajor
#define CM gl_CooperativeMatrixLayoutColumnMajor
#ifdef RR_GFX11
#include "rr_gfx11_k32.glsl"
#else
#define LDA(dst, base, k, mt, kt) coopMatLoad((dst), w8.w, (base) + 256u * ((mt) * ((k) / 16u) + (kt)), 16u, RM)
#define LDB(dst, base, n, kt, nt) coopMatLoad((dst), w8.w, (base) + 256u * ((kt) * ((n) / 16u) + (nt)), 16u, RM)
#define RND(m) [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (m)[i_] = rnd16((m)[i_]);
@@ -39,6 +46,7 @@ layout(set = 0, binding = 2, std430) buffer Arena { uint w[]; } arena;
#define K32(acc, a0, b0, a1, b1) { (acc) = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), (acc))); RND(acc) }
#define K32H(acc, a0, b0, a1, b1) { ACC p_ = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), ACC(acc))); [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (acc)[i_] = float16_t(p_[i_]); }
#endif
#endif
const uint HEAD = W_WK1 - W_WK0, BRANCH = W_W1_1 - W_W1_0;
shared uint KVS[2 * 768]; // the current head's K, then V^T; before the attention the key pixels' features
#define XF KVS // the key pixels' 32 FP8 features (plane 0 | plane 1), slot-major
@@ -55,10 +63,12 @@ ivec2 keyXY(uint t, int x0, int y0)
const int i = int(t & 7u), side = int((t - 64u) >> 3u);
return ivec2(side < 2 ? x0 + i : (side == 2 ? x0 - 1 : x0 + 8), side == 0 ? y0 - 1 : (side == 1 ? y0 + 8 : y0 + i));
}
#ifndef RR_GFX11
uint pack4(floate4m3_t a, floate4m3_t b, floate4m3_t c, floate4m3_t d)
{
return code(a) | (code(b) << 8u) | (code(c) << 16u) | (code(d) << 24u);
}
#endif
uvec2 mergeXY(uint r) { return r < 8u ? uvec2(r >> 1u, r & 1u) : uvec2((r - 8u) >> 1u, 2u + (r & 1u)); }
// the translated prologue's shared buffer sh: input_noise (10 x 10 pixels, 4 f16) at byte 0, input_tensor (two planes
// of 10 x 10 pixels x 16 FP8 features) at byte 800
engine/rr/enc0_m_tail.glsl
+56 -12
View File
@@ -23,15 +23,20 @@
RAC R[2];
for (uint n = wv; n < 6u; n += 4u) {
B8 fb0, fb1, hb_[2];
coopMatLoad(fb0, XF, n * 128u, 8u, CM);
coopMatLoad(fb1, XF, n * 128u + 4u, 8u, CM);
LDS8B(fb0, XF, n * 128u, 8u);
LDS8B(fb1, XF, n * 128u + 4u, 8u);
[[unroll]] for (uint m = 0u; m < 2u; ++m) {
RAC a;
[[unroll]] for (int i = 0; i < 8; ++i) a[i] = RV(w16.w[W_B1 / 2u + 16u * m + 8u * hl + uint(i)]);
A8 a0, a1;
LDA(a0, W_W1, 32u, m, 0u); LDA(a1, W_W1, 32u, m, 1u);
K32(a, a0, fb0, a1, fb1)
#ifdef RR_GFX11
{ F8V hc_ = uvec2(0u); [[unroll]] for (int i = 0; i < 8; ++i) hc_[i >> 2] |= q8s(max(float(a[i]), 0.0)) << (8u * uint(i & 3));
G11_B(hb_[m], hc_) }
#else
[[unroll]] for (int i = 0; i < 8; ++i) hb_[m][i] = q8s(max(float(a[i]), 0.0));
#endif
}
float16_t xv[2][8];
[[unroll]] for (uint m = 0u; m < 2u; ++m) {
@@ -67,13 +72,25 @@
precise float16_t se = s + float16_t(0.0001220703125);
const float16_t r = rsq16(se);
[[unroll]] for (uint m = 0u; m < 2u; ++m) {
#ifdef RR_GFX11
F8V xq = uvec2(0u);
#else
B8 xq;
#endif
[[unroll]] for (int i = 0; i < 8; ++i) {
precise float16_t g = r * w16.w[W_GAIN0 / 2u + inter(16u * m + 8u * hl + uint(i))];
precise float16_t nn = xv[m][i] * g;
#ifdef RR_GFX11
xq[i >> 2] |= q8h(nn) << (8u * uint(i & 3));
#else
xq[i] = q8h(nn);
#endif
}
#ifdef RR_GFX11
G11_ST(xq, XT, n * 128u + 4u * m, 8u)
#else
coopMatStore(xq, XT, n * 128u + 4u * m, 8u, CM);
#endif
}
}
barrier(); // XT complete; XF is read (the attention reuses its memory)
@@ -85,21 +102,17 @@
A8 wk[2], xa[2];
B8 wvb[2], xb[2];
[[unroll]] for (uint kt = 0u; kt < 2u; ++kt) {
coopMatLoad(xb[kt], XT, n * 128u + 4u * kt, 8u, CM);
coopMatLoad(xa[kt], XT, n * 128u + 4u * kt, 8u, RM);
LDS8B(xb[kt], XT, n * 128u + 4u * kt, 8u);
LDS8A(xa[kt], XT, n * 128u + 4u * kt, 8u);
LDA(wk[kt], W_WK0 + HEAD * hd, 32u, c, kt);
LDB(wvb[kt], W_WV0 + HEAD * hd, 32u, kt, c);
}
RAC a = RAC(RV(float16_t(0.0)));
K32(a, wk[0], xb[0], wk[1], xb[1])
A8 ka;
Q8(ka, a)
coopMatStore(ka, KVS, n * 128u + 4u * c, 8u, RM);
Q8STA(a, KVS, n * 128u + 4u * c, 8u)
RAC vv = RAC(RV(float16_t(0.0)));
K32(vv, xa[0], wvb[0], xa[1], wvb[1])
A8 va;
Q8(va, vv)
coopMatStore(va, KVS, 768u + c * 384u + 4u * n, 24u, RM);
Q8STA(vv, KVS, 768u + c * 384u + 4u * n, 24u)
}
barrier();
#include "rr_attn.glsl"
@@ -111,6 +124,9 @@
}
// MLP (dec0_m.comp's)
B8 Nb[2];
#ifdef RR_GFX11
F8V NbC[2] = F8V[2](uvec2(0u), uvec2(0u));
#endif
{
float16_t rv[2][8];
[[unroll]] for (uint m = 0u; m < 2u; ++m) [[unroll]] for (int i = 0; i < 8; ++i) rv[m][i] = float16_t(R[m][i]);
@@ -137,10 +153,17 @@
const uint k = 16u * m + 8u * hl + uint(i);
precise float16_t g = r * w16.w[W_GAIN1 / 2u + k];
precise float16_t nn = rv[m][i] * g;
#ifdef RR_GFX11
NbC[m][i >> 2] |= q8h(nn) << (8u * uint(i & 3));
#else
Nb[m][i] = q8h(nn);
#endif
precise float16_t a = rv[m][i] + w16.w[W_TABLE1 / 2u + k];
R[m][i] = RV(a);
}
#ifdef RR_GFX11
[[unroll]] for (uint m = 0u; m < 2u; ++m) G11_B(Nb[m], NbC[m])
#endif
}
for (uint br = 0u; br < 4u; ++br) {
const uint w1 = W_W1_0 + BRANCH * br, b1 = W_B1_0 + BRANCH * br, w2 = W_W2_0 + BRANCH * br;
@@ -151,7 +174,11 @@
A8 a0, a1;
LDA(a0, w1, 32u, c, 0u); LDA(a1, w1, 32u, c, 1u);
K32(h, a0, Nb[0], a1, Nb[1])
#ifdef RR_GFX11
{ F8V hc_; ACTQ8C(hc_, h) G11_B(Hb[c], hc_) }
#else
ACTQ8(Hb[c], h)
#endif
}
[[unroll]] for (uint m = 0u; m < 2u; ++m) {
A8 a0, a1;
@@ -171,8 +198,15 @@
const uint o = (outOff + uint(ty * ex + tx) * 32u) >> 2u;
barrier(); // XT is read; the merge rows reuse it
[[unroll]] for (uint m = 0u; m < 2u; ++m) {
#ifdef RR_GFX11
F8V tc_;
G11_CODES(tc_, R[m])
G11_B(Tb[m], tc_)
const uint q0 = tc_.x, q1 = tc_.y;
#else
Q8(Tb[m], R[m])
const uint q0 = pack4(Tb[m][0], Tb[m][1], Tb[m][2], Tb[m][3]), q1 = pack4(Tb[m][4], Tb[m][5], Tb[m][6], Tb[m][7]);
#endif
const uint at = 4u * m + 2u * hl;
if (inside) { arena.w[o + at] = q0; arena.w[o + at + 1u] = q1; }
const uint gm = mr * 32u + blk * 8u + 4u * m + 2u * hl;
@@ -232,11 +266,20 @@
[[unroll]] for (int i = 0; i < 8; ++i) a[i] = RV(w16.w[W_BM / 2u + 16u * m + 8u * hl + uint(i)]);
[[unroll]] for (uint ks = 0u; ks < 4u; ++ks) {
B8 g0, g1; A8 w0, w1;
coopMatLoad(g0, MG, 8u * ks, 32u, CM);
coopMatLoad(g1, MG, 8u * ks + 4u, 32u, CM);
LDS8B(g0, MG, 8u * ks, 32u);
LDS8B(g1, MG, 8u * ks + 4u, 32u);
LDA(w0, W_WM, 128u, m, 2u * ks); LDA(w1, W_WM, 128u, m, 2u * ks + 1u);
K32(a, w0, g0, w1, g1)
}
#ifdef RR_GFX11
F8V q;
G11_CODES(q, a)
if (X >= 0 && Y >= 0 && X < hx && Y < hy) {
const uint at = (mrgOff + (m >> 1u) * lplane + uint(Y * hx + X) * 32u) / 4u + 4u * (m & 1u) + 2u * hl;
arena.w[at] = q.x;
arena.w[at + 1u] = q.y;
}
#else
B8 q;
Q8(q, a)
if (X >= 0 && Y >= 0 && X < hx && Y < hy) {
@@ -244,6 +287,7 @@
arena.w[at] = pack4(q[0], q[1], q[2], q[3]);
arena.w[at + 1u] = pack4(q[4], q[5], q[6], q[7]);
}
#endif
}
if (wv == 3u && lane < 16u) { // surface 128: 2x2 sums of 0.25 * surface 112, rightwards first, then downwards
const uint mx = lane & 3u, my = lane >> 2u;
engine/rr/ptx2rr.py
+90 -4
View File
@@ -256,7 +256,17 @@ uint e4m3_to_h(uint b)
float f = e == 0u ? ldexp(float(m), -9) : ldexp(float(8u + m), int(max(e, 1u)) - 10);
return s | HU(float16_t(f));
}
#ifdef SOFTWARE_E4M3
#if defined(RR_GFX11)
// RDNA3 (engine/rr/rr_gfx11.glsl): ZLUDA's gfx11 conversions exactly. e4m3 -> f16: NaN -> 0x7fff with the sign
uint g11_e4m3_to_h(uint c)
{
const uint m = c & 0x7fu;
uint h = m >= 8u ? (m << 7u) + 0x2000u : uint(float16BitsToUint16(float16_t(float(m & 7u) * 0.001953125)));
h = m == 0x7fu ? 0x7fffu : h;
return ((c & 0x80u) << 8u) | h;
}
f16vec2 e4m3x2_to_f16x2(uint v) { return H2(g11_e4m3_to_h(v & 0xffu) | (g11_e4m3_to_h((v >> 8) & 0xffu) << 16)); }
#elif defined(SOFTWARE_E4M3)
f16vec2 e4m3x2_to_f16x2(uint v) { return H2(e4m3_to_h(v & 0xffu) | (e4m3_to_h((v >> 8) & 0xffu) << 16)); }
#else
// the hardware conversion: exact for every non-NaN code (2026-10-12); NaN keeps e4m3_to_h's sign | 0x7e00
@@ -284,7 +294,31 @@ uint h2e4m3(float f, bool relu)
else { int p = findMSB(n); bits = (uint(qe + p + 7) << 3) | ((n >> uint(p - 3)) & 7u); }
return s | min(bits, 0x7eu);
}
#ifdef SOFTWARE_E4M3
#if defined(RR_GFX11)
// f16 -> e4m3 as ZLUDA computes it on gfx11 (HIP's satfinite conversion, infinities clamped first): nearest even,
// |x| > 448 -> 0x7e, NaN -> 0x7f, the sign kept; .relu first zeroes negative non-NaN halves
uint g11_h2e4m3(uint h)
{
const uint s = (h >> 8u) & 0x80u, a = h & 0x7fffu;
if (a > 0x7c00u) return s | 0x7fu;
uint c;
if (a >= 0x2400u) {
const uint t = a - 0x2000u;
c = min((t + 0x3fu + ((t >> 7u) & 1u)) >> 7u, 0x7eu);
} else {
c = uint(roundEven(float(uint16BitsToFloat16(uint16_t(a))) * 512.0));
}
return s | c;
}
#define G11_HAVE_H2E4M3
uint g11_relu(uint h) { return (h & 0x8000u) != 0u && (h & 0x7fffu) <= 0x7c00u ? 0u : h; }
uint f16x2_to_e4m3x2(f16vec2 h, bool relu)
{
uint x = HU(h.x), y = HU(h.y);
if (relu) { x = g11_relu(x); y = g11_relu(y); }
return g11_h2e4m3(x) | (g11_h2e4m3(y) << 8);
}
#elif defined(SOFTWARE_E4M3)
uint f16x2_to_e4m3x2(f16vec2 h, bool relu) { return h2e4m3(float(h.x), relu) | (h2e4m3(float(h.y), relu) << 8); }
#else
// cvt.rn.satfinite(.relu).e4m3x2.f16x2 on the hardware conversion: round to nearest even after clamping to +-448
@@ -305,10 +339,38 @@ uint f16x2_to_e4m3x2(f16vec2 h, bool relu) { return h2e4m3_hw(h.x, relu) | (h2e4
// place (A row-major 16x32, B column-major 32x8 in a 16-column tile, C/D row-major f16 16x16), then loaded as
// whole matrices. Lane l = 4g + t holds A rows g and g+8 (k 4t.. and 16+4t..), B column g (same k), C/D row g
// and g+8, columns 2t and 2t+1.
#define MMA coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
#define MMB coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseB>
#define MMF coopmat<float, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#define MMH coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#ifdef RR_GFX11
// RDNA3: f16 operands holding ZLUDA's widening (value * 2^-8, k positions 2q + j <- 4q + j, 8 + 2q + j <- 4q + 2 + j
// in each K half), C scaled by 2^-16 into the two products and back (engine/rr/rr_gfx11.glsl)
#define MMA coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
#define MMB coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseB>
uvec2 g11_sc4(uint packed)
{
uint mag = packed & 0x7f7f7f7fu;
mag |= (mag + 0x01010101u) & 0x80808080u;
const uint sign = packed & 0x80808080u;
return uvec2(((mag & 0xffu) << 7u) | ((mag & 0xff00u) << 15u) | ((sign & 0x80u) << 8u) | ((sign & 0x8000u) << 16u),
(((mag >> 16u) & 0xffu) << 7u) | ((mag >> 24u) << 23u) | ((sign & 0x800000u) >> 8u) | (sign & 0x80000000u));
}
#define G11_FILL(m, w0, w1, w2, w3) { \
const uvec2 s0_ = g11_sc4(w0), s1_ = g11_sc4(w1), s2_ = g11_sc4(w2), s3_ = g11_sc4(w3); f16vec2 l_; \
l_ = unpackFloat2x16(s0_.x); (m)[0] = l_.x; (m)[1] = l_.y; l_ = unpackFloat2x16(s0_.y); (m)[8] = l_.x; (m)[9] = l_.y; \
l_ = unpackFloat2x16(s1_.x); (m)[2] = l_.x; (m)[3] = l_.y; l_ = unpackFloat2x16(s1_.y); (m)[10] = l_.x; (m)[11] = l_.y; \
l_ = unpackFloat2x16(s2_.x); (m)[4] = l_.x; (m)[5] = l_.y; l_ = unpackFloat2x16(s2_.y); (m)[12] = l_.x; (m)[13] = l_.y; \
l_ = unpackFloat2x16(s3_.x); (m)[6] = l_.x; (m)[7] = l_.y; l_ = unpackFloat2x16(s3_.y); (m)[14] = l_.x; (m)[15] = l_.y; }
#define G11_HAVE_FILL
#define coopMatLoad8(m, arr, off, stride) { const uint b_ = (off) + (gl_SubgroupInvocationID & 15u) * (stride); \
G11_FILL(m, arr[b_], arr[b_ + 1u], arr[b_ + 2u], arr[b_ + 3u]) }
#define MM8(x1, y1, x0, y0, c) MMF acc_ = MMF(c); for (int i_ = 0; i_ < acc_.length(); ++i_) acc_[i_] *= 1.52587890625e-05; \
acc_ = coopMatMulAdd(x1, y1, coopMatMulAdd(x0, y0, acc_)); for (int i_ = 0; i_ < acc_.length(); ++i_) acc_[i_] *= 65536.0;
#else
#define MMA coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
#define MMB coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseB>
#define coopMatLoad8(m, arr, off, stride) coopMatLoad(m, arr, off, stride, gl_CooperativeMatrixLayoutRowMajor)
#define MM8(x1, y1, x0, y0, c) MMF acc_ = coopMatMulAdd(x1, y1, coopMatMulAdd(x0, y0, MMF(c)));
#endif
void mma_e4m3_f16(uint a0, uint a1, uint a2, uint a3, uint b0, uint b1, uint c0, uint c1, out uint d0, out uint d1)
{
const uint base = gl_SubgroupID * 384u, A = base, B = base + 128u, C = base + 256u;
@@ -323,6 +385,16 @@ void mma_e4m3_f16(uint a0, uint a1, uint a2, uint a3, uint b0, uint b1, uint c0,
mms[C + g * 8u + 4u + t] = 0u; mms[C + (g + 8u) * 8u + 4u + t] = 0u;
memoryBarrierShared(); subgroupBarrier();
MMA x0, x1; MMB y0, y1; MMH c;
#ifdef RR_GFX11
coopMatLoad8(x0, mms, A, 8u);
coopMatLoad8(x1, mms, A + 4u, 8u);
coopMatLoad8(y0, mms, B, 8u);
coopMatLoad8(y1, mms, B + 4u, 8u);
coopMatLoad(c, mms, C, 8u, gl_CooperativeMatrixLayoutRowMajor);
MM8(x1, y1, x0, y0, c)
memoryBarrierShared(); subgroupBarrier();
coopMatStore(MMH(acc_), mms, C, 8u, gl_CooperativeMatrixLayoutRowMajor);
#else
coopMatLoad(x0, mms, A, 8u, gl_CooperativeMatrixLayoutRowMajor);
coopMatLoad(x1, mms, A + 4u, 8u, gl_CooperativeMatrixLayoutRowMajor);
coopMatLoad(y0, mms, B, 8u, gl_CooperativeMatrixLayoutColumnMajor);
@@ -331,6 +403,7 @@ void mma_e4m3_f16(uint a0, uint a1, uint a2, uint a3, uint b0, uint b1, uint c0,
MMF acc = coopMatMulAdd(x1, y1, coopMatMulAdd(x0, y0, MMF(c)));
memoryBarrierShared(); subgroupBarrier();
coopMatStore(MMH(acc), mms, C, 8u, gl_CooperativeMatrixLayoutRowMajor);
#endif
memoryBarrierShared(); subgroupBarrier();
d0 = mms[C + g * 8u + t]; d1 = mms[C + (g + 8u) * 8u + t];
}
@@ -350,6 +423,16 @@ void mma_e4m3_f16_pair(uint a0, uint a1, uint a2, uint a3, uint b0, uint b1, uin
mms[C + g * 8u + 4u + t] = f0; mms[C + (g + 8u) * 8u + 4u + t] = f1;
memoryBarrierShared(); subgroupBarrier();
MMA x0, x1; MMB y0, y1; MMH c;
#ifdef RR_GFX11
coopMatLoad8(x0, mms, A, 8u);
coopMatLoad8(x1, mms, A + 4u, 8u);
coopMatLoad8(y0, mms, B, 8u);
coopMatLoad8(y1, mms, B + 4u, 8u);
coopMatLoad(c, mms, C, 8u, gl_CooperativeMatrixLayoutRowMajor);
MM8(x1, y1, x0, y0, c)
memoryBarrierShared(); subgroupBarrier();
coopMatStore(MMH(acc_), mms, C, 8u, gl_CooperativeMatrixLayoutRowMajor);
#else
coopMatLoad(x0, mms, A, 8u, gl_CooperativeMatrixLayoutRowMajor);
coopMatLoad(x1, mms, A + 4u, 8u, gl_CooperativeMatrixLayoutRowMajor);
coopMatLoad(y0, mms, B, 8u, gl_CooperativeMatrixLayoutColumnMajor);
@@ -358,6 +441,7 @@ void mma_e4m3_f16_pair(uint a0, uint a1, uint a2, uint a3, uint b0, uint b1, uin
MMF acc = coopMatMulAdd(x1, y1, coopMatMulAdd(x0, y0, MMF(c)));
memoryBarrierShared(); subgroupBarrier();
coopMatStore(MMH(acc), mms, C, 8u, gl_CooperativeMatrixLayoutRowMajor);
#endif
memoryBarrierShared(); subgroupBarrier();
d0 = mms[C + g * 8u + t]; d1 = mms[C + (g + 8u) * 8u + t];
h0 = mms[C + g * 8u + 4u + t]; h1 = mms[C + (g + 8u) * 8u + 4u + t];
@@ -803,7 +887,9 @@ class RRT(base.T):
w('#extension GL_KHR_shader_subgroup_ballot : require')
w('#extension GL_KHR_cooperative_matrix : require')
w('#extension GL_KHR_memory_scope_semantics : require')
w('#ifndef RR_GFX11')
w('#extension GL_EXT_float_e4m3 : require')
w('#endif')
w('#extension GL_EXT_spirv_intrinsics : require')
w(f'// Generated by engine/rr/ptx2rr.py from {self.kernel}. Model data derived from the user\'s DLL; do not commit.')
w(f'layout(local_size_x = {self.bx * self.by * self.bz}, local_size_y = 1, local_size_z = 1) in;')
engine/rr/rr_attn.glsl
+27 -6
View File
@@ -13,17 +13,23 @@
#define RR_VS0 0u
#endif
B8 Qb[2];
[[unroll]] for (uint c = 0u; c < 2u; ++c) coopMatLoad(Qb[c], RR_KS, RR_KS0 + wv * 128u + 4u * c, 8u, CM);
[[unroll]] for (uint c = 0u; c < 2u; ++c) LDS8B(Qb[c], RR_KS, RR_KS0 + wv * 128u + 4u * c, 8u);
// scores S^T (keys x this wave's queries) + bias, then the cubic softmax down each column
B8 Pb[6];
{
float16_t e[6][8];
[[unroll]] for (uint kt = 0u; kt < 6u; ++kt) {
HAC c;
#ifdef RR_GFX11
// the accumulator layout the stages are written for (rr_gfx11.glsl): element i is row i + 8 hl
[[unroll]] for (int i = 0; i < 8; ++i)
c[i] = w16.w[(W_BIAS0 + HEAD * hd) / 2u + kt * 1024u + 16u * wv + (8u * hl + uint(i)) * 64u + col];
#else
coopMatLoad(c, w16.w, (W_BIAS0 + HEAD * hd) / 2u + kt * 1024u + 16u * wv, 64u, RM);
#endif
A8 k0, k1;
coopMatLoad(k0, RR_KS, RR_KS0 + kt * 128u, 8u, RM);
coopMatLoad(k1, RR_KS, RR_KS0 + kt * 128u + 4u, 8u, RM);
LDS8A(k0, RR_KS, RR_KS0 + kt * 128u, 8u);
LDS8A(k1, RR_KS, RR_KS0 + kt * 128u + 4u, 8u);
K32H(c, k0, Qb[0], k1, Qb[1])
// the cubic numerator from enc1_pack.cubic_table, indexed by the score's binary16 bits
[[unroll]] for (int i = 0; i < 8; ++i) e[kt][i] = hb(uint(w16u.w[W_CUBIC / 2u + bh(c[i])]));
@@ -49,12 +55,23 @@
// reciprocal (within an ulp) already rounds correctly
const float16_t r = float16_t(1.0 / float(rs));
const f16vec2 r2 = f16vec2(r, r);
[[unroll]] for (uint kt = 0u; kt < 6u; ++kt)
[[unroll]] for (uint kt = 0u; kt < 6u; ++kt) {
#ifdef RR_GFX11
F8V pc_ = uvec2(0u);
#endif
[[unroll]] for (uint q = 0u; q < 4u; ++q) {
precise f16vec2 pv = ep[kt][q] * r2;
#ifdef RR_GFX11
pc_[q >> 1u] |= (q8h(pv.x) | (q8h(pv.y) << 8u)) << (16u * (q & 1u));
#else
// a probability is in [0, 1]: the plain conversion, no saturation clamp, both in one instruction
{ const fe4m3vec2 t_ = fe4m3vec2(pv); Pb[kt][2u * q] = t_.x; Pb[kt][2u * q + 1u] = t_.y; }
#endif
}
#ifdef RR_GFX11
G11_B(Pb[kt], pc_)
#endif
}
}
// O^T = V^T P^T, then the out-projection onto the residual
B8 Ob[2];
@@ -62,9 +79,13 @@
RAC o = RAC(RV(float16_t(0.0)));
[[unroll]] for (uint ks = 0u; ks < 3u; ++ks) {
A8 v0, v1;
coopMatLoad(v0, RR_VS, RR_VS0 + c * 384u + 8u * ks, 24u, RM);
coopMatLoad(v1, RR_VS, RR_VS0 + c * 384u + 8u * ks + 4u, 24u, RM);
LDS8A(v0, RR_VS, RR_VS0 + c * 384u + 8u * ks, 24u);
LDS8A(v1, RR_VS, RR_VS0 + c * 384u + 8u * ks + 4u, 24u);
K32(o, v0, Pb[2u * ks], v1, Pb[2u * ks + 1u])
}
#ifdef RR_GFX11
{ F8V oc_; G11_CODES(oc_, o) G11_B(Ob[c], oc_) }
#else
Q8(Ob[c], o)
#endif
}
engine/rr/rr_exact.glsl
+32 -1
View File
@@ -1,5 +1,6 @@
// Exact NVIDIA arithmetic helpers for the native Ray Reconstruction stages (engine/rr/*_m.comp, enc1_native.comp).
// Requires GL_EXT_shader_explicit_arithmetic_types, GL_EXT_float_e4m3.
// Requires GL_EXT_shader_explicit_arithmetic_types, GL_EXT_float_e4m3 (RR_GFX11: GL_KHR_shader_subgroup_shuffle instead;
// FP8 values are then codes, rr_gfx11.glsl).
//
// RADV folds a register-only float(float16_t(x)) back to x, so every binary16 rounding whose result is used again goes
// through a conversion with an opaque zero (RR_ZERO16, set in main from something the compiler cannot evaluate, e.g.
@@ -30,6 +31,20 @@ uint inter(uint j) { return (j & ~31u) | (((j & 31u) >> 2u) << 1u) | (j & 1u) |
uint uninter(uint c) { return (c & ~31u) | (((c & 15u) >> 1u) << 2u) | (c & 1u) | (((c >> 4u) & 1u) << 1u); }
int mirror(int v, int n) { v = abs(v); return min(v, 2 * n - 2 - v); }
#ifdef RR_GFX11
#include "rr_gfx11.glsl"
#define F8 uint
uint q8h(float16_t x) { return g11_code(x); }
uint q8s(float x) { return g11_codef(x); }
uint code(uint c) { return c; }
float16_t e4h(uint c) { return g11_e4h(c); }
// q8(act(h)) from the packed table (enc1_pack.act_table; the weight buffer's W8U view at W_ACT)
#define ACTQ(h) uint(w8u.w[W_ACT + bh(h)])
// an accumulator's activation codes (accumulator layout)
#define ACTQ8C(dst, h) { (dst) = uvec2(0u); [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (dst)[i_ >> 2] |= ACTQ(float16_t((h)[i_])) << (8u * uint(i_ & 3)); }
#else
#define F8 floate4m3_t
float16_t e4h(floate4m3_t v) { return float16_t(v); }
// cvt.rn.satfinite.e4m3 (NaN -> 0x7f)
floate4m3_t q8(float16_t h)
{
@@ -41,6 +56,7 @@ floate4m3_t q8s(float x) { floate4m3_t r; saturatedConvertEXT(r, x); return r; }
floate4m3_t q8h(float16_t x) { floate4m3_t r; saturatedConvertEXT(r, x); return r; }
uint code(floate4m3_t v) { return uint(floate4m3BitsToUintEXT(v)); }
#endif
float16_t hb(uint b) { return uint16BitsToFloat16(uint16_t(b)); }
uint bh(float16_t h) { return uint(float16BitsToUint16(h)); }
@@ -106,6 +122,7 @@ float16_t fma16q(float16_t a, float16_t b, float16_t c, inout bool mid)
return hb(bh(float16_t(s)) | uint(RR_ZERO16));
#endif
}
#ifndef RR_GFX11
// The saturating FP8 conversion of a whole accumulator, two elements per conversion instruction
#define Q8PAIRS(dst, src) [[unroll]] for (int i_ = 0; i_ < 8; i_ += 2) { fe4m3vec2 t_; saturatedConvertEXT(t_, vec2((src)[i_], (src)[i_ + 1])); (dst)[i_] = t_.x; (dst)[i_ + 1] = t_.y; }
// q8(act(h)) from the packed table (enc1_pack.act_table; the weight buffer's W8U view at W_ACT)
@@ -117,6 +134,7 @@ float16_t fma16q(float16_t a, float16_t b, float16_t c, inout bool mid)
#else
#define ACTQ8(dst, h) [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (dst)[i_] = ACTQ(float16_t((h)[i_]));
#endif
#endif
float16_t act(float16_t x)
{
precise float16_t y = clamp(x, float16_t(-2.0), float16_t(2.0));
@@ -152,3 +170,16 @@ float16_t cubic_e(float16_t x)
const float16_t u = fma16(m, t, float16_t(1.375));
return hb((bh(u) << 5u) & 0x7FE0u);
}
// FP8 operands from a shared byte image (word offset and stride): an A row-major, a B column-major; and an accumulator
// quantized and stored as the transposed A row-major / as a B column-major (the same bytes)
#ifdef RR_GFX11
#define LDS8A(m, ARR, off, stride) LD8A(m, ARR, off, stride)
#define LDS8B(m, ARR, off, stride) LD8B(m, ARR, off, stride)
#define Q8STA(src, ARR, off, stride) { F8V c_; G11_CODES(c_, src) G11_ST(c_, ARR, off, stride) }
#define Q8STB(src, ARR, off, stride) Q8STA(src, ARR, off, stride)
#else
#define LDS8A(m, ARR, off, stride) coopMatLoad(m, ARR, off, stride, gl_CooperativeMatrixLayoutRowMajor)
#define LDS8B(m, ARR, off, stride) coopMatLoad(m, ARR, off, stride, gl_CooperativeMatrixLayoutColumnMajor)
#define Q8STA(src, ARR, off, stride) { A8 q; Q8(q, src) coopMatStore(q, ARR, off, stride, gl_CooperativeMatrixLayoutRowMajor); }
#define Q8STB(src, ARR, off, stride) { B8 q; Q8(q, src) coopMatStore(q, ARR, off, stride, gl_CooperativeMatrixLayoutColumnMajor); }
#endif
engine/rr/rr_gfx11.glsl
+95
View File
@@ -0,0 +1,95 @@
// RDNA3 (gfx11, RR_GFX11) forms of the native Ray Reconstruction stages' FP8 operations, included by rr_exact.glsl.
// gfx11 has no FP8 matrix instruction and RADV no VK_EXT_shader_float8, so the reference is ZLUDA's gfx11 lowering of
// mma.m16n8k32.f16.e4m3.e4m3.f16 (D4R_ZLUDA_WMMA_FP8, patches/zluda 0004), which these reproduce bit for bit:
// - each e4m3 byte is widened on its bits to the binary16 of value * 2^-8 (magnitude at bits 7..14, the sign at 15,
// the NaN code to a NaN), so the products carry 2^-16;
// - the accumulator enters scaled by 2^-16, the two K halves (k 0..15, 16..31) run as two chained
// v_wmma_f32_16x16x16_f16, the sum is scaled back and rounded to binary16 once;
// - inside each K half the operand positions hold k in ZLUDA's order: position 2q + j holds k 4q + j and position
// 8 + 2q + j holds k 4q + 2 + j (q < 4, j < 2). The instruction's f32 sum depends on that order (tests on an
// RX 7700 XT: 29% of the outputs change when the same products are permuted), so it is part of the arithmetic.
// FP8 values in registers are their codes (uint); an operand matrix is coopmat<float16_t> holding the widened values.
// RADV's gfx11 layouts: an A or B operand lane l holds row / column l & 15, element i = k i (both half-waves alike);
// an accumulator lane holds column l & 15, element i = row 2i + (l >> 4). The stages' elementwise code is written for
// gfx12's accumulator layout (element i = row i + 8 (l >> 4)), so every A operand is loaded with its rows permuted:
// lane l takes row g11_row(l & 15), and a product's physical row 2i + h is then row i + 8h, i.e. the gfx12 layout.
// That only relabels the outputs (each is its own dot product), so it is exact. Accumulators loaded or built
// elementwise therefore use the gfx12 positions too.
#define A8 coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
#define B8 coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseB>
// cvt.rn.satfinite.e4m3 of binary16 bits as ZLUDA computes it on gfx11 (HIP's conversion after clamping infinities
// to 448): nearest even, |x| > 448 -> 0x7e, NaN -> 0x7f, the sign kept. Equal to it for all 65536 inputs.
#ifndef G11_HAVE_H2E4M3 // ptx2rr.py's preamble (a translated head before this file) defines the same
uint g11_h2e4m3(uint h)
{
const uint s = (h >> 8u) & 0x80u, a = h & 0x7fffu;
if (a > 0x7c00u) return s | 0x7fu;
uint c;
if (a >= 0x2400u) { // |x| >= 2^-6: an e4m3 normal; rebias 15 -> 7 and round 10 -> 3 mantissa bits
const uint t = a - 0x2000u;
c = min((t + 0x3fu + ((t >> 7u) & 1u)) >> 7u, 0x7eu);
} else { // e4m3 subnormals: steps of 2^-9 (x * 2^9 is exact in f32)
c = uint(roundEven(float(uint16BitsToFloat16(uint16_t(a))) * 512.0));
}
return s | c;
}
#endif
uint g11_code(float16_t h) { return g11_h2e4m3(uint(float16BitsToUint16(h))); }
// an f32 that holds a binary16 value
uint g11_codef(float x) { return g11_code(float16_t(x)); }
// the exact binary16 value of a code (ZLUDA's e4m3_to_f16_bits; NaN -> 0x7fff with the sign)
float16_t g11_e4h(uint c)
{
const uint m = c & 0x7fu;
uint h = m >= 8u ? (m << 7u) + 0x2000u : uint(float16BitsToUint16(float16_t(float(m & 7u) * 0.001953125)));
h = m == 0x7fu ? 0x7fffu : h;
return uint16BitsToFloat16(uint16_t(((c & 0x80u) << 8u) | h));
}
#ifndef G11_HAVE_FILL
// four codes -> the scaled binary16 pairs (bytes 0, 1) and (bytes 2, 3), ZLUDA's e4m3x4_to_scaled_f16x2x2
uvec2 g11_sc4(uint packed)
{
uint mag = packed & 0x7f7f7f7fu;
mag |= (mag + 0x01010101u) & 0x80808080u;
const uint sign = packed & 0x80808080u;
const uint lo = ((mag & 0xffu) << 7u) | ((mag & 0xff00u) << 15u) | ((sign & 0x80u) << 8u) | ((sign & 0x8000u) << 16u);
const uint hi = (((mag >> 16u) & 0xffu) << 7u) | ((mag >> 24u) << 23u) | ((sign & 0x800000u) >> 8u) | (sign & 0x80000000u);
return uvec2(lo, hi);
}
// an operand from its lane's 16 codes in k order (four words): positions in ZLUDA's order
#define G11_FILL(m, w0, w1, w2, w3) { \
const uvec2 s0_ = g11_sc4(w0), s1_ = g11_sc4(w1), s2_ = g11_sc4(w2), s3_ = g11_sc4(w3); f16vec2 l_; \
l_ = unpackFloat2x16(s0_.x); (m)[0] = l_.x; (m)[1] = l_.y; l_ = unpackFloat2x16(s0_.y); (m)[8] = l_.x; (m)[9] = l_.y; \
l_ = unpackFloat2x16(s1_.x); (m)[2] = l_.x; (m)[3] = l_.y; l_ = unpackFloat2x16(s1_.y); (m)[10] = l_.x; (m)[11] = l_.y; \
l_ = unpackFloat2x16(s2_.x); (m)[4] = l_.x; (m)[5] = l_.y; l_ = unpackFloat2x16(s2_.y); (m)[12] = l_.x; (m)[13] = l_.y; \
l_ = unpackFloat2x16(s3_.x); (m)[6] = l_.x; (m)[7] = l_.y; l_ = unpackFloat2x16(s3_.y); (m)[14] = l_.x; (m)[15] = l_.y; }
#endif
// the A operand row a lane holds: physical row r carries row (r >> 1) + 8 (r & 1)
uint g11_row(uint r) { return (r >> 1u) | ((r & 1u) << 3u); }
// an A operand row-major (rows permuted) or a B operand column-major from a shared byte image (word offset and
// stride, as coopMatLoad takes them for an FP8 matrix): the lane's row / column is 16 consecutive bytes
#define LD8A(m, ARR, off, stride) { const uint b_ = (off) + g11_row(gl_SubgroupInvocationID & 15u) * (stride); \
G11_FILL(m, ARR[b_], ARR[b_ + 1u], ARR[b_ + 2u], ARR[b_ + 3u]) }
#define LD8B(m, ARR, off, stride) { const uint b_ = (off) + (gl_SubgroupInvocationID & 15u) * (stride); \
G11_FILL(m, ARR[b_], ARR[b_ + 1u], ARR[b_ + 2u], ARR[b_ + 3u]) }
// natural k order (a matrix loaded from the widened weights) -> ZLUDA's positions
#define G11_PERM(d, s) { (d)[0] = (s)[0]; (d)[1] = (s)[1]; (d)[2] = (s)[4]; (d)[3] = (s)[5]; (d)[4] = (s)[8]; \
(d)[5] = (s)[9]; (d)[6] = (s)[12]; (d)[7] = (s)[13]; (d)[8] = (s)[2]; (d)[9] = (s)[3]; (d)[10] = (s)[6]; \
(d)[11] = (s)[7]; (d)[12] = (s)[10]; (d)[13] = (s)[11]; (d)[14] = (s)[14]; (d)[15] = (s)[15]; }
// FP8 vectors in the accumulator layout: element i of a lane (row i + 8 half) as byte i & 3 of word i >> 2
#define F8V uvec2
// the codes of an accumulator that holds binary16 values
#define G11_CODES(dst, src) { (dst) = uvec2(0u); [[unroll]] for (int i_ = 0; i_ < 8; ++i_) \
(dst)[i_ >> 2] |= g11_codef((src)[i_]) << (8u * uint(i_ & 3)); }
// the B operand that is the accumulator-layout codes c (the gfx12 elementwise conversion of an accumulator into B):
// rows 0..7 are half-wave 0's codes, 8..15 half-wave 1's
#define G11_B(dst, c) { const uvec2 o_ = uvec2(subgroupShuffleXor((c).x, 16u), subgroupShuffleXor((c).y, 16u)); \
const bool h_ = gl_SubgroupInvocationID >= 16u; const uvec2 e0_ = h_ ? o_ : (c), e1_ = h_ ? (c) : o_; \
G11_FILL(dst, e0_.x, e0_.y, e1_.x, e1_.y) }
// store the codes c column-major as bytes (column j at word off + j * stride): what an FP8 B stored column-major or
// an FP8 A (the accumulator's transpose) stored row-major leaves; each half-wave writes its 8 rows
#define G11_ST(c, ARR, off, stride) { const uint b_ = (off) + (gl_SubgroupInvocationID & 15u) * (stride) + \
(gl_SubgroupInvocationID >= 16u ? 2u : 0u); ARR[b_] = (c).x; ARR[b_ + 1u] = (c).y; }
engine/rr/rr_gfx11_k32.glsl
+29
View File
@@ -0,0 +1,29 @@
// RR_GFX11: the weight loads and the 32-deep step of the native stages (rr_gfx11.glsl explains the arithmetic).
// Included after ACC / HAC / RM / CM and the binding-1 views. W_WIDE (a -D of the build) is the f16 element at which
// the weight file carries its widened copy: element W_WIDE + j is byte j of the image widened like an FP8 operand
// (value * 2^-8), so a stage's byte offsets address it unchanged.
layout(set = 0, binding = 1, std430) readonly buffer W128 { uvec4 w[]; } w128;
#ifndef W_WIDE
#error RR_GFX11 needs W_WIDE (the widened weight copy, engine/rr/widen_weights.py)
#endif
// A: the lane's (permuted) row is 16 consecutive f16, two 16-byte loads (base and tiles are multiples of 16 bytes)
#define LDA(dst, base, k, mt, kt) { const uint e_ = W_WIDE + (base) + 256u * ((mt) * ((k) / 16u) + (kt)) + \
16u * g11_row(gl_SubgroupInvocationID & 15u); const uvec4 u0_ = w128.w[e_ >> 3u], u1_ = w128.w[(e_ >> 3u) + 1u]; \
f16vec2 l_; l_ = unpackFloat2x16(u0_.x); (dst)[0] = l_.x; (dst)[1] = l_.y; l_ = unpackFloat2x16(u0_.y); (dst)[8] = l_.x; (dst)[9] = l_.y; \
l_ = unpackFloat2x16(u0_.z); (dst)[2] = l_.x; (dst)[3] = l_.y; l_ = unpackFloat2x16(u0_.w); (dst)[10] = l_.x; (dst)[11] = l_.y; \
l_ = unpackFloat2x16(u1_.x); (dst)[4] = l_.x; (dst)[5] = l_.y; l_ = unpackFloat2x16(u1_.y); (dst)[12] = l_.x; (dst)[13] = l_.y; \
l_ = unpackFloat2x16(u1_.z); (dst)[6] = l_.x; (dst)[7] = l_.y; l_ = unpackFloat2x16(u1_.w); (dst)[14] = l_.x; (dst)[15] = l_.y; }
#define LDB(dst, base, n, kt, nt) { B8 t_; coopMatLoad(t_, w16.w, W_WIDE + (base) + 256u * ((kt) * ((n) / 16u) + (nt)), 16u, RM); G11_PERM(dst, t_) }
#define RND(m) [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (m)[i_] = rnd16((m)[i_]);
#if defined(RR_MIX) || defined(RR_RELAXED)
#error RR_GFX11 builds the exact arithmetic only
#endif
#define RAC ACC
#define RV(x) float(x)
// C * 2^-16 (exact), the two halves, * 2^16 (exact), one rounding to binary16
#define K32(acc, a0, b0, a1, b1) { [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (acc)[i_] *= 1.52587890625e-05; \
(acc) = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), (acc))); \
[[unroll]] for (int i_ = 0; i_ < 8; ++i_) (acc)[i_] = rnd16((acc)[i_] * 65536.0); }
#define K32H(acc, a0, b0, a1, b1) { ACC p_ = ACC(acc); [[unroll]] for (int i_ = 0; i_ < 8; ++i_) p_[i_] *= 1.52587890625e-05; \
p_ = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), p_)); \
[[unroll]] for (int i_ = 0; i_ < 8; ++i_) (acc)[i_] = float16_t(p_[i_] * 65536.0); }
engine/rr/swin_m.comp
+61 -11
View File
@@ -12,7 +12,9 @@
#extension GL_EXT_shader_explicit_arithmetic_types : require
#extension GL_EXT_shader_16bit_storage : require
#extension GL_EXT_shader_8bit_storage : require
#ifndef RR_GFX11
#extension GL_EXT_float_e4m3 : require
#endif
#extension GL_KHR_shader_subgroup_basic : require
#extension GL_KHR_shader_subgroup_shuffle : require
#extension GL_EXT_control_flow_attributes : require
@@ -26,7 +28,9 @@ layout(local_size_x = 128) in;
#define NB (C / 8)
layout(set = 0, binding = 0, std430) readonly buffer ParamBlock { uint p[]; } param;
#ifndef RR_GFX11
layout(set = 0, binding = 1, std430) readonly buffer W8 { floate4m3_t w[]; } w8;
#endif
layout(set = 0, binding = 1, std430) readonly buffer W16 { float16_t w[]; } w16;
layout(set = 0, binding = 1, std430) readonly buffer W16U { uint16_t w[]; } w16u;
layout(set = 0, binding = 1, std430) readonly buffer W8U { uint8_t w[]; } w8u;
@@ -46,10 +50,15 @@ layout(set = 0, binding = 6, rgba16f) uniform writeonly image2D S128;
#define ACC coopmat<float, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#define HAC coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#ifndef RR_GFX11
#define A8 coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
#define B8 coopmat<floate4m3_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseB>
#endif
#define RM gl_CooperativeMatrixLayoutRowMajor
#define CM gl_CooperativeMatrixLayoutColumnMajor
#ifdef RR_GFX11
#include "rr_gfx11_k32.glsl"
#else
#define LDA(dst, base, k, mt, kt) coopMatLoad((dst), w8.w, (base) + 256u * ((mt) * ((k) / 16u) + (kt)), 16u, RM)
#define LDB(dst, base, n, kt, nt) coopMatLoad((dst), w8.w, (base) + 256u * ((kt) * ((n) / 16u) + (nt)), 16u, RM)
#define RND(m) [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (m)[i_] = rnd16((m)[i_]);
@@ -71,6 +80,7 @@ layout(set = 0, binding = 6, rgba16f) uniform writeonly image2D S128;
#define K32(acc, a0, b0, a1, b1) { (acc) = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), (acc))); RND(acc) }
#define K32H(acc, a0, b0, a1, b1) { ACC p_ = coopMatMulAdd((a1), (b1), coopMatMulAdd((a0), (b0), ACC(acc))); [[unroll]] for (int i_ = 0; i_ < 8; ++i_) (acc)[i_] = float16_t(p_[i_]); }
#endif
#endif
const uint HEAD = W_WK1 - W_WK0, BRANCH = W_W1_1 - W_W1_0;
// NVIDIA's reduction over a short list (swin_ref.pairs_tree)
@@ -124,11 +134,16 @@ ivec2 keyXY(uint t, int x0, int y0)
return ivec2(side < 2 ? x0 + i : (side == 2 ? x0 - 1 : x0 + 8), side == 0 ? y0 - 1 : (side == 1 ? y0 + 8 : y0 + i));
}
int clampTap(int v, int n) { return v < 0 ? 1 : (v >= n ? n - 2 : v); }
#ifdef RR_GFX11
uint pack4(uint a, uint b, uint c, uint d) { return a | (b << 8u) | (c << 16u) | (d << 24u); }
float16_t e4(uint w, uint b) { return g11_e4h((w >> (8u * b)) & 0xffu); }
#else
uint pack4(floate4m3_t a, floate4m3_t b, floate4m3_t c, floate4m3_t d)
{
return code(a) | (code(b) << 8u) | (code(c) << 16u) | (code(d) << 24u);
}
float16_t e4(uint w, uint b) { return float16_t(uintBitsToFloate4m3EXT(uint8_t((w >> (8u * b)) & 0xffu))); }
#endif
uvec2 mergeXY(uint r) { return r < 8u ? uvec2(r >> 1u, r & 1u) : uvec2((r - 8u) >> 1u, 2u + (r & 1u)); }
void main()
@@ -156,7 +171,7 @@ void main()
barrier();
[[unroll]] for (uint n = 0u; n < 3u; ++n) {
B8 lb[CX / 16];
[[unroll]] for (uint kt = 0u; kt < CX / 16; ++kt) coopMatLoad(lb[kt], POOL, LOW_AT + n * 4u * CX + 4u * kt, CX / 4u, CM);
[[unroll]] for (uint kt = 0u; kt < CX / 16; ++kt) LDS8B(lb[kt], POOL, LOW_AT + n * 4u * CX + 4u * kt, CX / 4u);
for (uint mm = 0u; mm < MT; ++mm) {
const uint m = MT * wv + mm;
RAC a;
@@ -166,9 +181,7 @@ void main()
LDA(w0, W_WE, CX, m, kt); LDA(w1, W_WE, CX, m, kt + 1u);
K32(a, w0, lb[kt], w1, lb[kt + 1u])
}
B8 q;
Q8(q, a)
coopMatStore(q, EX, n * 16u * C + 4u * m, C, CM);
Q8STB(a, EX, n * 16u * C + 4u * m, C)
}
}
barrier();
@@ -216,7 +229,7 @@ void main()
precise float16_t se = sm + float16_t(0.0001220703125);
const float16_t rr = rsq16(se);
[[unroll]] for (uint wd = 0u; wd < C / 4; ++wd) {
floate4m3_t c[4];
F8 c[4];
[[unroll]] for (uint b = 0u; b < 4u; ++b) {
const uint j = 4u * wd + b;
precise float16_t g = rr * w16.w[W_GAIN0 / 2u + inter(j)];
@@ -265,20 +278,25 @@ void main()
[[unroll]] for (uint kt = 0u; kt < 2u * KC; kt += 2u) {
B8 xb0, xb1, wv0, wv1;
A8 xa0, xa1, wk0, wk1;
coopMatLoad(xb0, XTS, XT_AT + n * 4u * C + 4u * kt, C / 4u, CM);
coopMatLoad(xb1, XTS, XT_AT + n * 4u * C + 4u * kt + 4u, C / 4u, CM);
coopMatLoad(xa0, XTS, XT_AT + n * 4u * C + 4u * kt, C / 4u, RM);
coopMatLoad(xa1, XTS, XT_AT + n * 4u * C + 4u * kt + 4u, C / 4u, RM);
LDS8B(xb0, XTS, XT_AT + n * 4u * C + 4u * kt, C / 4u);
LDS8B(xb1, XTS, XT_AT + n * 4u * C + 4u * kt + 4u, C / 4u);
LDS8A(xa0, XTS, XT_AT + n * 4u * C + 4u * kt, C / 4u);
LDS8A(xa1, XTS, XT_AT + n * 4u * C + 4u * kt + 4u, C / 4u);
LDA(wk0, W_WK0 + HEAD * hd, C, c, kt); LDA(wk1, W_WK0 + HEAD * hd, C, c, kt + 1u);
LDB(wv0, W_WV0 + HEAD * hd, 32u, kt, c); LDB(wv1, W_WV0 + HEAD * hd, 32u, kt + 1u, c);
K32(a, wk0, xb0, wk1, xb1)
K32(v, xa0, wv0, xa1, wv1)
}
#ifdef RR_GFX11
Q8STA(a, RR_KS, RR_KS0 + n * 128u + 4u * c, 8u)
Q8STA(v, RR_VS, RR_VS0 + c * 384u + 4u * n, 24u)
#else
A8 ka, va;
Q8(ka, a)
Q8(va, v)
coopMatStore(ka, RR_KS, RR_KS0 + n * 128u + 4u * c, 8u, RM);
coopMatStore(va, RR_VS, RR_VS0 + c * 384u + 4u * n, 24u, RM);
#endif
}
barrier();
#include "rr_attn.glsl"
@@ -291,6 +309,10 @@ void main()
// ---------------------------------------------------------------- MLP
B8 Nb[MT];
#ifdef RR_GFX11
F8V NbC[MT];
[[unroll]] for (uint m = 0u; m < MT; ++m) NbC[m] = uvec2(0u);
#endif
{
float16_t rv[MT][8];
[[unroll]] for (uint m = 0u; m < MT; ++m) [[unroll]] for (int i = 0; i < 8; ++i) rv[m][i] = float16_t(R[m][i]);
@@ -326,10 +348,17 @@ void main()
const uint k = 16u * m + 8u * hl + uint(i);
precise float16_t g = r * w16.w[W_GAIN14 / 2u + k];
precise float16_t n = rv[m][i] * g;
#ifdef RR_GFX11
NbC[m][i >> 2] |= q8h(n) << (8u * uint(i & 3));
#else
Nb[m][i] = q8h(n);
#endif
precise float16_t a = rv[m][i] + w16.w[W_TABLE16 / 2u + k];
R[m][i] = RV(a);
}
#ifdef RR_GFX11
[[unroll]] for (uint m = 0u; m < MT; ++m) G11_B(Nb[m], NbC[m])
#endif
}
[[dont_unroll]] for (uint br = 0u; br < NB; ++br) {
const uint w1 = W_W1_0 + BRANCH * br, b1 = W_B1_0 + BRANCH * br, w2 = W_W2_0 + BRANCH * br;
@@ -342,7 +371,11 @@ void main()
LDA(a0, w1, C, c, kt); LDA(a1, w1, C, c, kt + 1u);
K32(h, a0, Nb[kt], a1, Nb[kt + 1u])
}
#ifdef RR_GFX11
{ F8V hc_; ACTQ8C(hc_, h) G11_B(Hb[c], hc_) }
#else
ACTQ8(Hb[c], h)
#endif
}
[[unroll]] for (uint m = 0u; m < MT; ++m) {
A8 a0, a1;
@@ -365,8 +398,15 @@ void main()
barrier(); // every wave is done with XT (the merge rows reuse it)
#endif
[[unroll]] for (uint m = 0u; m < MT; ++m) {
#ifdef RR_GFX11
F8V tc_;
G11_CODES(tc_, R[m])
G11_B(Tb[m], tc_)
const uint w0 = tc_.x, w1 = tc_.y;
#else
Q8(Tb[m], R[m])
const uint w0 = pack4(Tb[m][0], Tb[m][1], Tb[m][2], Tb[m][3]), w1 = pack4(Tb[m][4], Tb[m][5], Tb[m][6], Tb[m][7]);
#endif
const uint at = 4u * (m & 1u) + 2u * hl;
if (inside) { arena.w[o + (m >> 1u) * (plane >> 2u) + at] = w0; arena.w[o + (m >> 1u) * (plane >> 2u) + at + 1u] = w1; }
#if KIND == 0 && CX
@@ -447,11 +487,20 @@ void main()
[[unroll]] for (int i = 0; i < 8; ++i) a[i] = RV(w16.w[W_BM / 2u + 16u * m + 8u * hl + uint(i)]);
for (uint ks = 0u; ks < C / 8u; ++ks) {
B8 g0, g1; A8 w0, w1;
coopMatLoad(g0, MG, 8u * ks, C, CM);
coopMatLoad(g1, MG, 8u * ks + 4u, C, CM);
LDS8B(g0, MG, 8u * ks, C);
LDS8B(g1, MG, 8u * ks + 4u, C);
LDA(w0, W_WM, 4u * C, m, 2u * ks); LDA(w1, W_WM, 4u * C, m, 2u * ks + 1u);
K32(a, w0, g0, w1, g1)
}
#ifdef RR_GFX11
F8V q;
G11_CODES(q, a)
if (X >= 0 && Y >= 0 && X < hx && Y < hy) {
const uint at = o + (m >> 1u) * (lplane >> 2u) + 4u * (m & 1u) + 2u * hl;
arena.w[at] = q.x;
arena.w[at + 1u] = q.y;
}
#else
B8 q;
Q8(q, a)
if (X >= 0 && Y >= 0 && X < hx && Y < hy) {
@@ -459,6 +508,7 @@ void main()
arena.w[at] = pack4(q[0], q[1], q[2], q[3]);
arena.w[at + 1u] = pack4(q[4], q[5], q[6], q[7]);
}
#endif
}
if (wv == 3u && lane < 16u) { // surface 128: 2x2 sums of 0.25 * surface 112, rightwards first, then downwards
const uint mx = lane & 3u, my = lane >> 2u;
engine/rr/widen_weights.py
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env python3
"""Append the RDNA3 (RR_GFX11) widened copy of a native stage's weight file and print its f16 element offset (W_WIDE).
usage: widen_weights.py WEIGHTS.bin [OUT.bin] (default: in place)
Element W_WIDE + j of the file's f16 view is byte j of the original image as ZLUDA widens an FP8 operand on gfx11
(rr_gfx11.glsl): the magnitude at bits 7..14, the sign at bit 15, the NaN code with bit 14 set too, i.e. the binary16
of value * 2^-8. Bytes that are not FP8 (tables, binary16 data) are widened as well and never read. W_WIDE is the
original size rounded up to 512 bytes, halved.
"""
import sys
from pathlib import Path
import numpy as np
def widen(image):
b = np.frombuffer(image, np.uint8).astype(np.uint16)
mag = b & 0x7f
mag = mag | ((mag + 1) & 0x80)
return ((mag << 7) | ((b & 0x80) << 8)).astype('<u2').tobytes()
def main():
src = Path(sys.argv[1]); dst = Path(sys.argv[2]) if len(sys.argv) > 2 else src
image = src.read_bytes()
base = (len(image) + 511) // 512 * 512
dst.write_bytes(image + bytes(base - len(image)) + widen(image))
print(base // 2)
if __name__ == '__main__':
main()
engine/rr_camera.h
+63
View File
@@ -0,0 +1,63 @@
// Preset D/E camera transforms in NGX 310.7's single-precision operation order.
// Keeping the inversions and reprojection products matters: their near-zero residues feed specular motion history.
#pragma once
#include <array>
#include <cmath>
namespace d4r::rr_camera {
using Matrix = std::array<float, 16>;
inline Matrix multiply(const Matrix& a, const Matrix& b) {
Matrix r{};
for (int i = 0; i < 4; ++i)
for (int j = 0; j < 4; ++j) {
float v = a[4 * i] * b[j];
v += a[4 * i + 1] * b[4 + j];
v += a[4 * i + 2] * b[8 + j];
v += a[4 * i + 3] * b[12 + j];
r[4 * i + j] = v;
}
return r;
}
inline Matrix inverse(const Matrix& m) {
Matrix r{};
float determinant = 0;
for (int i = 0; i < 4; ++i) {
determinant = 0;
for (int j = 0; j < 4; ++j) {
float t[9]; int n = 0;
for (int row = 0; row < 4; ++row)
if (row != i)
for (int col = 0; col < 4; ++col)
if (col != j) t[n++] = m[4 * row + col];
float cofactor = (t[4] * t[8] - t[7] * t[5]) * t[0];
cofactor -= (t[3] * t[8] - t[6] * t[5]) * t[1];
cofactor += (t[3] * t[7] - t[6] * t[4]) * t[2];
cofactor *= ((i + j) & 1) ? -1.0f : 1.0f;
r[4 * j + i] = cofactor;
determinant += cofactor * m[4 * i + j];
}
}
// The final row's determinant and one division, followed by sixteen multiplies, match the reference.
if (determinant == 0 || !std::isfinite(determinant))
return Matrix{1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1};
const float reciprocal = 1.0f / determinant;
for (float& v : r) v *= reciprocal;
return r;
}
struct Transforms { Matrix previousClip, inverseClip, inversePixels, previousPixels; };
inline Transforms transforms(const Matrix& view, const Matrix& clip, const Matrix& previousView,
const Matrix& previousClip, float width, float height) {
Matrix camera = inverse(view), previousCamera = inverse(previousView);
for (int i = 12; i < 15; ++i) {
previousCamera[i] -= camera[i];
camera[i] = 0.0f;
}
const Matrix current = multiply(inverse(camera), clip);
const Matrix previous = multiply(inverse(previousCamera), previousClip);
const Matrix inverted = inverse(current);
const Matrix pixels{width * 0.5f, 0, 0, 0, 0, height * -0.5f, 0, 0,
0, 0, 1, 0, width * 0.5f, height * 0.5f, 0, 1};
return {previous, inverted, inverse(multiply(current, pixels)),
multiply(multiply(current, multiply(inverted, previous)), pixels)};
}
} // namespace d4r::rr_camera
engine/runtime.cpp
+1
View File
@@ -1,5 +1,6 @@
#include "runtime.h"
#include "vk_dispatch.h"
#include "rr_camera.h"
#include <algorithm>
#include <cmath>
#include <cstdlib>
engine/runtime_rr.inc
+21
View File
@@ -76,6 +76,7 @@ void putMat(std::vector<uint8_t>& p, uint32_t offset, const Mat4& m) {
struct RModel::Impl {
Device device;
uint32_t preset = 0;
bool gfx11 = false;
std::unique_ptr<R2Model> v2; // preset F ("D4RR0002"): engine/runtime_rr2.inc
// per stage, the shader's bindings after the three buffers (engine/rr/README.md): texture argument offsets, then
// surface argument offsets with their GLSL image format, both ascending
@@ -119,6 +120,11 @@ RModel::RModel(VkPhysicalDevice physical, VkDevice device, const std::string& di
std::istringstream f(line);
std::string key; f >> key;
if (key == "preset") f >> m.preset;
else if (key == "arithmetic") {
std::string arithmetic; f >> arithmetic;
if (arithmetic != "gfx11") throw std::runtime_error("rr.txt: unknown arithmetic " + arithmetic);
m.gfx11 = true;
}
else if (key == "stage") {
std::string name, shader, weightFile, textureList, surfaceList, option; uint32_t subgroup = 32;
f >> name >> shader >> subgroup >> weightFile >> textureList >> surfaceList >> option;
@@ -256,6 +262,7 @@ struct REngine::Impl {
std::vector<Bindings> bindings;
const Bindings* bound = nullptr;
float prevJitter[2] = {0, 0};
rr_camera::Matrix previousView{}, previousClip{};
uint32_t frame = 0;
bool reset = true;
explicit Impl(RModel::Impl& m) : model(m) {}
@@ -443,6 +450,20 @@ void REngine::setFrame(const RFrameImages& images, const RFrameSettings& setting
putMat(p, 596, rc); putMat(p, 660, clip); putMat(p, 724, clip); putMat(p, 788, inv);
putMat(p, 852, mul(toNdc, inv)); putMat(p, 916, mul(rc, toPixels));
putF(p, 756, settings.jitter[0] * (2.0f / rw)); putF(p, 760, -settings.jitter[1] * (2.0f / rh));
if (e.model.gfx11) {
rr_camera::Matrix v, c;
for (int i = 0; i < 16; ++i) { v[i] = settings.worldToView[i]; c[i] = settings.viewToClip[i]; }
const auto camera = rr_camera::transforms(v, c, reset ? v : e.previousView,
reset ? c : e.previousClip, rw, rh);
auto store = [&](uint32_t offset, const rr_camera::Matrix& matrix) {
for (int i = 0; i < 16; ++i) putF(p, offset + 4 * i, matrix[i]);
};
store(596, camera.previousClip); store(788, camera.inverseClip);
store(852, camera.inversePixels); store(916, camera.previousPixels);
putF(p, 756, c[8] + (settings.jitter[0] + settings.jitter[0]) / rw);
putF(p, 760, c[9] - (settings.jitter[1] * 2.0f) / rh);
e.previousView = v; e.previousClip = c;
}
put32(p, 984, f);
put32(p, 1024, e.rw); put32(p, 1028, e.rh);
// the output stage
kernels/support/fsr_rr_compose.comp
+55
View File
@@ -0,0 +1,55 @@
#version 450
// FSR Ray Regeneration backend, stage 3: the denoised diffuse and specular signals remodulated by
// their albedos, plus the residual the denoiser never saw (particles, what demodulation could not
// carry, sky). The result is render-resolution colour for the Super Sampling stage.
layout(local_size_x = 8, local_size_y = 8) in;
layout(binding = 0, rgba16f) uniform readonly image2D denoisedDiffuse;
layout(binding = 1, rgba16f) uniform readonly image2D denoisedSpecular;
layout(binding = 2, rgba16f) uniform readonly image2D diffuseAlbedo;
layout(binding = 3, rgba16f) uniform readonly image2D specularAlbedo;
layout(binding = 4, rgba16f) uniform readonly image2D residual;
layout(binding = 5, rgba16f) uniform writeonly image2D composed;
layout(binding = 6, rgba16f) uniform readonly image2D noisyDiffuse;
layout(binding = 7, rgba16f) uniform readonly image2D noisySpecular;
layout(push_constant) uniform Constants
{
uvec2 renderSize;
float haloLimit; // > 0: each denoised lobe at most this multiple of the brightest noisy input of that lobe around it
uint unused;
} pc;
void main()
{
const ivec2 px = ivec2(gl_GlobalInvocationID.xy);
if (px.x >= int(pc.renderSize.x) || px.y >= int(pc.renderSize.y))
return;
const vec3 da = imageLoad(diffuseAlbedo, px).rgb, sa = imageLoad(specularAlbedo, px).rgb;
vec3 diffuse = imageLoad(denoisedDiffuse, px).rgb * da;
vec3 specular = imageLoad(denoisedSpecular, px).rgb * sa;
if (pc.haloLimit > 0.0)
{
// The denoiser can carry a thin object's large demodulated signal (dark albedo) across its edge, where this
// texel's albedo turns it into a bright fringe. A denoised value above everything around it in the noisy
// input is such a leak: capped, by luminance (the hue is kept).
const vec3 w = vec3(0.2126, 0.7152, 0.0722);
float diffuseMax = 0.0, specularMax = 0.0;
for (int dy = -1; dy <= 1; ++dy)
for (int dx = -1; dx <= 1; ++dx)
{
const ivec2 q = clamp(px + ivec2(dx, dy), ivec2(0), ivec2(pc.renderSize) - 1);
// what the lobe contributes there (the neighbour's own albedo): a leaked demodulated signal
// remodulated by this texel's albedo exceeds all of these
diffuseMax = max(diffuseMax, dot(imageLoad(noisyDiffuse, q).rgb * imageLoad(diffuseAlbedo, q).rgb, w));
specularMax = max(specularMax, dot(imageLoad(noisySpecular, q).rgb * imageLoad(specularAlbedo, q).rgb, w));
}
const float dl = dot(diffuse, w), sl = dot(specular, w);
if (dl > pc.haloLimit * diffuseMax)
diffuse *= pc.haloLimit * diffuseMax / dl;
if (sl > pc.haloLimit * specularMax)
specular *= pc.haloLimit * specularMax / sl;
}
const vec3 color = diffuse + specular + imageLoad(residual, px).rgb;
imageStore(composed, px, vec4(clamp(color, 0.0, 65000.0), 1.0));
}
kernels/support/fsr_rr_prepare.comp
+194
View File
@@ -0,0 +1,194 @@
#version 450
// FSR Ray Regeneration backend, stage 1: a game's DLSS Ray Reconstruction inputs -> the inputs of
// AMD's FidelityFX denoiser 1.2 (ffx_denoiser.h), at render resolution.
//
// The split of the game's single noisy colour into a diffuse and a specular signal follows the
// DLSS-RR -> FSR-RR adapter of Zach Hembree's OptiScaler branch ffx-denoise-experimental (GPL-3.0,
// 18ac686, FSRDInputConv.hlsl, "Mode 2") and burak113's continuation (5a6adb1): each lobe gets the
// colour in proportion to its albedo and is demodulated by it. That is an approximation - one
// colour cannot be split exactly - and what does not survive demodulation goes to the residual,
// which bypasses the denoiser. Unlike that reference this writes FSR-RR 1.2's contract: signed
// linear depth reconstructed from hardware or linear depth, motion as previous-minus-current UV,
// linear (non-gamma) albedo and the colour before particles as the denoised base.
layout(local_size_x = 8, local_size_y = 8) in;
layout(binding = 0) uniform sampler2D inColor;
layout(binding = 1) uniform sampler2D inDepth;
layout(binding = 2) uniform sampler2D inMotion;
layout(binding = 3) uniform sampler2D inNormals;
layout(binding = 4) uniform sampler2D inRoughness; // the normals again when roughness is packed
layout(binding = 5) uniform sampler2D inDiffuseAlbedo;
layout(binding = 6) uniform sampler2D inSpecularAlbedo;
layout(binding = 7) uniform sampler2D inSpecularHit; // bound to the normals when absent
layout(binding = 8) uniform sampler2D inDiffuseHit; // bound to the normals when absent
layout(binding = 9) uniform sampler2D inBeforeParticles; // bound to the colour when absent
layout(binding = 10, r32f) uniform writeonly image2D outLinearDepth;
layout(binding = 11, rgba16f) uniform writeonly image2D outMotion;
layout(binding = 12, rgba16f) uniform writeonly image2D outNormals;
layout(binding = 13, rgba16f) uniform writeonly image2D outDiffuseAlbedo;
layout(binding = 14, rgba16f) uniform writeonly image2D outSpecularAlbedo;
layout(binding = 15, rgba16f) uniform writeonly image2D outDiffuse;
layout(binding = 16, rgba16f) uniform writeonly image2D outSpecular;
layout(binding = 17, rgba16f) uniform writeonly image2D outResidual;
// Matrices are uploaded as the row-major, row-vector (v' = vM) float[16] NGX and FFX use. Read as a
// GLSL (column-major) mat4 that is the transpose, so M * v here is the row-vector product v * M.
layout(push_constant) uniform Constants
{
mat4 clipToView; // inverse(ViewToClipMatrix)
mat4 viewToPreviousView; // inverse(WorldToView) * previous WorldToView
vec4 renderSize; // width, height, 1 / width, 1 / height
vec4 motion; // xy: MV texel -> UV delta (MV.Scale / MV resolution); zw: MV texel per render pixel
uvec4 colorDepthBase; // colour subrect origin xy, depth subrect origin xy
uvec4 motionBaseFlags; // motion subrect origin xy, flags, unused
vec4 depthParams; // x: sign of view z in front of the camera, y: sky depth (hardware), z: far marker
vec4 detail; // x: share of the colour that bypasses the denoiser (0: none); y: smallest albedo a
// signal is divided by
} pc;
const uint kHardwareDepth = 1u;
const uint kRoughnessPacked = 2u;
const uint kSpecularHit = 4u;
const uint kSpecularHitInW = 8u; // DLSSD.SpecularRayDirectionHitDistance: the distance is .w
const uint kDiffuseHit = 16u;
const uint kDiffuseHitInW = 32u;
const uint kNormalsUnorm = 64u; // [0,1]-encoded normals (UNORM formats)
const uint kBeforeParticles = 128u;
const uint kInvalidNormalPassthrough = 256u;
const float kHalfMax = 65000.0;
vec3 safe(vec3 v)
{
// NaN fails both comparisons and becomes 0
return vec3(v.x >= 0.0 ? min(v.x, kHalfMax) : 0.0, v.y >= 0.0 ? min(v.y, kHalfMax) : 0.0,
v.z >= 0.0 ? min(v.z, kHalfMax) : 0.0);
}
// AMD's NormalToOctahedronUv (FidelityFX denoising.md 5.3.1)
vec2 octahedron(vec3 n)
{
n /= abs(n.x) + abs(n.y) + abs(n.z);
vec2 k = vec2(n.x >= 0.0 ? 1.0 : -1.0, n.y >= 0.0 ? 1.0 : -1.0);
vec2 xy = n.z < 0.0 ? (1.0 - abs(n.yx)) * k : n.xy;
return xy * 0.5 + 0.5;
}
void main()
{
const ivec2 px = ivec2(gl_GlobalInvocationID.xy);
if (px.x >= int(pc.renderSize.x) || px.y >= int(pc.renderSize.y))
return;
const uint flags = pc.motionBaseFlags.z;
const vec2 uv = (vec2(px) + 0.5) * pc.renderSize.zw;
const vec2 ndc = vec2(uv.x * 2.0 - 1.0, 1.0 - uv.y * 2.0);
// Signed view-space position. Hardware depth goes through the inverse projection, which carries
// the game's handedness and (reversed) depth range; linear depth is the distance along the view
// axis, given its sign by the projection.
const float depth = texelFetch(inDepth, ivec2(pc.colorDepthBase.zw) + px, 0).r;
vec3 viewPos;
bool sky;
if ((flags & kHardwareDepth) != 0u)
{
const vec4 p = pc.clipToView * vec4(ndc, depth, 1.0);
viewPos = p.xyz / p.w;
sky = depth == pc.depthParams.y || !(abs(p.w) > 0.0);
}
else
{
const vec4 p = pc.clipToView * vec4(ndc, 0.5, 1.0);
const vec3 ray = p.xyz / p.w;
const float z = depth * pc.depthParams.x;
viewPos = ray * (z / ray.z);
sky = !(depth > 0.0) || depth >= pc.depthParams.z;
}
const vec3 color = safe(texelFetch(inColor, ivec2(pc.colorDepthBase.xy) + px, 0).rgb);
const vec3 base = (flags & kBeforeParticles) != 0u ? safe(texelFetch(inBeforeParticles, px, 0).rgb) : color;
// Particles (colour after minus before) stay out of the denoiser; signed, so smoke can darken.
const vec3 particles = color - base;
vec3 diffuseAlbedo = clamp(texelFetch(inDiffuseAlbedo, px, 0).rgb, 0.0, 1.0);
vec3 specularAlbedo = clamp(texelFetch(inSpecularAlbedo, px, 0).rgb, 0.0, 1.0);
// Energy above 1 is taken from the diffuse lobe first (reference: albedo overshoot clamp).
diffuseAlbedo = max(diffuseAlbedo - max(diffuseAlbedo + specularAlbedo - 1.0, 0.0), 0.0);
const float albedoSum = dot(diffuseAlbedo + specularAlbedo, vec3(1.0));
sky = sky || !(albedoSum > 3e-3) || any(isnan(viewPos)) || any(isinf(viewPos));
const vec4 normalTexel = texelFetch(inNormals, px, 0);
vec3 normal = (flags & kNormalsUnorm) != 0u ? normalTexel.xyz * 2.0 - 1.0 : normalTexel.xyz;
const float normalLength = dot(normal, normal);
// A zero/invalid normal is no surface, not an invented world-Z-facing mirror. Alan Wake 2
// supplies zero normals for its sky with finite FP16-max depth, so the far bound alone misses it.
if ((flags & kInvalidNormalPassthrough) != 0u)
sky = sky || !(normalLength > 1e-8) || isnan(normalLength) || isinf(normalLength);
if (sky)
{
// Outside linearDepthBounds: the denoiser passes these texels through, and the residual carries
// the whole colour.
imageStore(outLinearDepth, px, vec4(pc.depthParams.z * 10.0 * pc.depthParams.x));
imageStore(outMotion, px, vec4(0.0));
imageStore(outNormals, px, vec4(0.5, 0.5, 1.0, 0.0));
imageStore(outDiffuseAlbedo, px, vec4(0.0));
imageStore(outSpecularAlbedo, px, vec4(0.0));
imageStore(outDiffuse, px, vec4(0.0));
imageStore(outSpecular, px, vec4(0.0));
imageStore(outResidual, px, vec4(color, 0.0));
return;
}
imageStore(outLinearDepth, px, vec4(viewPos.z));
// 2.5D motion: DLSS motion vectors point from this pixel to where it was (previous - current), in
// the MV texture's pixels. The depth delta is the camera's: the previous view's z of this point
// (moving geometry is not reprojected; there is no previous depth for it).
const ivec2 mvTexel = ivec2(pc.motionBaseFlags.xy) + ivec2(vec2(px) * pc.motion.zw);
const vec2 mv = texelFetch(inMotion, mvTexel, 0).xy * pc.motion.xy;
const vec4 previous = pc.viewToPreviousView * vec4(viewPos, 1.0);
imageStore(outMotion, px, vec4(mv, previous.z / previous.w - viewPos.z, 0.0));
normal = dot(normal, normal) > 1e-8 ? normalize(normal) : vec3(0.0, 0.0, 1.0);
const float roughness = clamp((flags & kRoughnessPacked) != 0u ? normalTexel.w : texelFetch(inRoughness, px, 0).r,
0.0, 1.0);
// material type 0 (the default material) everywhere
imageStore(outNormals, px, vec4(octahedron(normal), roughness, 0.0));
imageStore(outDiffuseAlbedo, px, vec4(diffuseAlbedo, 0.0));
imageStore(outSpecularAlbedo, px, vec4(specularAlbedo, 0.0));
// Split by albedo and demodulate (reference "Mode 2"). Per channel, so a tinted metal keeps its tint.
const vec3 total = diffuseAlbedo + specularAlbedo;
const vec3 specularShare = specularAlbedo / max(total, vec3(1e-4));
const vec3 specularColor = base * specularShare;
const vec3 diffuseColor = base - specularColor;
// A dark albedo would turn light it did not shape (fog, sky light, the background at a thin object's edge) into
// a huge signal that the denoiser clips (fog lost) or spreads (bright fringes): no albedo below the floor
// divides a signal, and what the division cannot carry stays in the residual.
vec3 diffuse = safe(diffuseColor / max(diffuseAlbedo, vec3(pc.detail.y)));
vec3 specular = safe(specularColor / max(specularAlbedo, vec3(pc.detail.y)));
if (pc.detail.x > 0.0)
{
// Fine detail the denoiser treats as noise (hair, water ripples): part of the colour goes around it,
// through the residual.
const float denoised = 1.0 - clamp(pc.detail.x, 0.0, 1.0);
diffuse *= denoised;
specular *= denoised;
}
const vec3 remodulated = diffuse * diffuseAlbedo + specular * specularAlbedo;
// A: the ray hit distance for an indirect signal (ignored, but non-negative, for a direct one)
float specularHit = 0.0, diffuseHit = 0.0;
if ((flags & kSpecularHit) != 0u)
{
const vec4 hit = texelFetch(inSpecularHit, px, 0);
specularHit = clamp((flags & kSpecularHitInW) != 0u ? hit.w : hit.x, 0.0, kHalfMax);
}
if ((flags & kDiffuseHit) != 0u)
{
const vec4 hit = texelFetch(inDiffuseHit, px, 0);
diffuseHit = clamp((flags & kDiffuseHitInW) != 0u ? hit.w : hit.x, 0.0, kHalfMax);
}
imageStore(outDiffuse, px, vec4(diffuse, diffuseHit));
imageStore(outSpecular, px, vec4(specular, specularHit));
imageStore(outResidual, px, vec4(base - remodulated + particles, 0.0));
}
kernels/support/motion_dilate.hip
+8
View File
@@ -62,3 +62,11 @@ void d4r_motion_dilate(D4rMotionDilationParams p)
}
*(uint32_t*)(output + (uint64_t)y * p.motion_pitch + 4 * x) = value;
}
#include "motion_dilate_lds.hip"
#include "native_motion_compose.hip"
#include "wire_history.hip"
#include "native_motion_prepared.hip"
kernels/support/motion_dilate_lds.hip
+164
View File
@@ -0,0 +1,164 @@
// Exact low-res, unjittered specialization. Compile as wave32 only.
#include "../../tools/d4r_motion_dilation.h"
#pragma clang fp contract(off)
#define D4R_LDS_SYNC() do{__builtin_amdgcn_fence(__ATOMIC_RELEASE,"workgroup");__builtin_amdgcn_s_barrier();__builtin_amdgcn_fence(__ATOMIC_ACQUIRE,"workgroup");}while(0)
#define LDS __attribute__((shared))
__attribute__((device)) static inline int clampi(int x, int n) { return x < 0 ? 0 : x >= n ? n - 1 : x; }
template<bool ThinSky>
__attribute__((device, always_inline)) static inline
void motion_dilate_lds(D4rMotionDilationParams p, uint32_t allocation_height)
{
LDS float tile[32][33];
LDS unsigned rowmask[32];
LDS unsigned anyfg;
LDS uint64_t extended[ThinSky ? 38 : 1];
const int t = __builtin_amdgcn_workitem_id_x();
const int tx = t & 15, ty = t >> 4;
const int ox = __builtin_amdgcn_workgroup_id_x() * 16 - 8, oy = __builtin_amdgcn_workgroup_id_y() * 16 - 8;
const int W = p.render_width, H = p.render_height;
const bool rev = p.reversed_depth;
const uint8_t* depth = (const uint8_t*)p.depth;
const uint8_t* motion = (const uint8_t*)p.motion;
uint8_t* output = (uint8_t*)p.output;
// Extra tiles and inactive lanes copy allocation padding, including pitch bytes.
// The bridge launches over pitch/4 by allocation_height, not the active grid.
const int x = ox + 8 + tx, y = oy + 8 + ty;
const bool allocated = (uint64_t)x < p.motion_pitch / 4 && y < (int)allocation_height;
if (allocated && (x >= W || y >= H))
*(uint32_t*)(output + (uint64_t)y * p.motion_pitch + 4 * x) =
*(const uint32_t*)(motion + (uint64_t)y * p.motion_pitch + 4 * x);
// This exit is uniform across the workgroup; partial tiles reach both barriers.
if (ox + 8 >= W || oy + 8 >= H) return;
if (t == 0) anyfg = 0;
#pragma unroll
for (int j = 0; j < 2; ++j)
#pragma unroll
for (int i = 0; i < 2; ++i)
{
const int ly = ty + 16 * j, lx = tx + 16 * i;
tile[ly][lx] = *(const float*)(depth + (uint64_t)clampi(oy + ly, H) * p.depth_pitch + 4 * clampi(ox + lx, W));
}
D4R_LDS_SYNC();
if constexpr (ThinSky)
{
// Source candidates reach the edge of the 32x32 tile. Three extra
// foreground bits on each side prove whether a candidate belongs to a
// run of four, without truncating runs at a workgroup boundary.
const int lane = t & 31, wave = t >> 5;
for (int r = wave; r < 38; r += 8)
{
const int row = r - 3, gy = oy + row;
bool fg[2];
#pragma unroll
for (int part = 0; part < 2; ++part)
{
const int col = lane - 3 + part * 32, gx = ox + col;
const bool valid = (part == 0 || lane < 6) && gx >= 0 && gx < W && gy >= 0 && gy < H;
float d = 0;
if (valid)
d = row >= 0 && row < 32 && col >= 0 && col < 32 ? tile[row][col] :
*(const float*)(depth + (uint64_t)gy * p.depth_pitch + 4 * gx);
fg[part] = valid && (rev ? d > 1.e-7f : d < 1.0f - 1.e-7f);
}
const unsigned low = __builtin_amdgcn_ballot_w32(fg[0]);
const unsigned high = __builtin_amdgcn_ballot_w32(fg[1]);
if (lane == 0) extended[r] = low | ((uint64_t)high << 32);
}
D4R_LDS_SYNC();
#pragma unroll
for (int r = 0; r < 4; ++r)
if (lane == 0)
{
const int row = wave * 4 + r + 3;
const uint64_t m = extended[row];
const uint64_t heads = m & (m >> 1) & (m >> 2) & (m >> 3);
const uint64_t horizontal = heads | (heads << 1) | (heads << 2) | (heads << 3);
uint64_t vertical = 0;
#pragma unroll
for (int k = -3; k <= 0; ++k)
vertical |= extended[row+k] & extended[row+k+1] & extended[row+k+2] & extended[row+k+3];
// Preserve long wires/poles: short in either cross-section.
const unsigned thin = (unsigned)((m & ~(horizontal & vertical)) >> 3);
rowmask[row-3] = thin;
if (thin) __atomic_fetch_or(&anyfg, 1u, __ATOMIC_RELAXED);
}
}
else
{
// Each wave32 handles four rows, one ballot per row. Out-of-image halo
// values are clamped for the opaque scan but excluded from the sky masks.
{
const int lane = t & 31, wave = t >> 5;
#pragma unroll
for (int r = 0; r < 4; ++r)
{
const int row = wave * 4 + r, gy = oy + row, gx = ox + lane;
const float d = tile[row][lane];
const bool fg = (rev ? d > 1.e-7f : d < 1.0f - 1.e-7f) && gy >= 0 && gy < H && gx >= 0 && gx < W;
const unsigned m = __builtin_amdgcn_ballot_w32(fg);
if (lane == 0) { rowmask[row] = m; if (m) __atomic_fetch_or(&anyfg, 1u, __ATOMIC_RELAXED); }
}
}
}
D4R_LDS_SYNC();
if (x >= W || y >= H) return;
const int lx = tx + 8, ly = ty + 8;
const float current = tile[ly][lx];
float best = current;
int bx = x, by = y;
const int R = (int)p.radius;
#pragma unroll
for (int dy = -2; dy <= 2; ++dy)
#pragma unroll
for (int dx = -2; dx <= 2; ++dx)
{
if (dy < -R || dy > R || dx < -R || dx > R) continue;
const float c = tile[ly + dy][lx + dx];
if (rev ? c > best : c < best)
best = c, bx = x + dx, by = y + dy;
}
bx = clampi(bx, W); by = clampi(by, H);
const bool sky = rev ? current <= 1.e-7f : current >= 1.0f - 1.e-7f;
bool sky_filled = false;
if (sky && anyfg)
{
const int wd[9] = {8, 7, 7, 7, 6, 6, 5, 3, 0};
int bd = 65, fx = 0, fy = 0;
#pragma unroll
for (int dy = -8; dy <= 8; ++dy)
{
const int w = wd[dy < 0 ? -dy : dy];
unsigned m = rowmask[ly + dy];
const int lo = lx - w, hi = lx + w;
m &= (hi >= 31 ? 0xffffffffu : ((1u << (hi + 1)) - 1u)) & (lo <= 0 ? 0xffffffffu : ~((1u << lo) - 1u));
if (!m) continue;
const unsigned left = m & ((2u << lx) - 1u), right = m & ~((1u << lx) - 1u);
int cdx = 99;
if (left) cdx = (31 - __builtin_clz(left)) - lx;
if (right) { const int b = __builtin_ctz(right) - lx; if (cdx == 99 || b < -cdx) cdx = b; }
const int d2 = cdx * cdx + dy * dy, nx = x + cdx;
if (d2 < bd || (d2 == bd && nx < fx)) bd = d2, fx = nx, fy = y + dy;
}
if (bd <= 64) bx = fx, by = fy, sky_filled = true;
}
uint32_t value = *(const uint32_t*)(motion + (uint64_t)y * p.motion_pitch + 4 * x);
const bool nearer = rev ? best > current * 1.02f + 1.e-7f : best < current * 0.98f - 1.e-7f;
if (sky_filled || nearer)
value = *(const uint32_t*)(motion + (uint64_t)by * p.motion_pitch + 4 * bx);
*(uint32_t*)(output + (uint64_t)y * p.motion_pitch + 4 * x) = value;
}
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256, 256)))
void d4r_motion_dilate_lds(D4rMotionDilationParams p, uint32_t allocation_height)
{
motion_dilate_lds<false>(p, allocation_height);
}
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256, 256)))
void d4r_motion_dilate_lds_thin(D4rMotionDilationParams p, uint32_t allocation_height)
{
motion_dilate_lds<true>(p, allocation_height);
}
#undef LDS
#undef D4R_LDS_SYNC
kernels/support/native_motion_compose.hip
+232
View File
@@ -0,0 +1,232 @@
// Complete AW2 native geometry + camera composition and output-grid dilation.
// Private prototype; input buffers are immutable and no NGX routing is changed.
#include "../../tools/d4r_native_motion.h"
#pragma clang fp contract(off)
__attribute__((device,always_inline)) static inline float d4r_native_project(const float* m,int c,float x,float y,float z)
{
float value=m[c]*x;
value=__builtin_fmaf(m[4+c],y,value);
value=__builtin_fmaf(m[8+c],z,value);
return value+m[12+c];
}
__attribute__((device,always_inline)) static inline void d4r_native_motion_finish(const D4rNativeMotionParams& p,
uint32_t x,uint32_t y,uint32_t bx,uint32_t by,float best)
{
const uint32_t geometry=*(const uint32_t*)((const uint8_t*)p.geometry+(uint64_t)by*p.geometry_pitch+4*bx);
const uint16_t lo=geometry,hi=geometry>>16;
const bool lo_nan=(lo&0x7c00)==0x7c00 && (lo&0x3ff);
const bool hi_nan=(hi&0x7c00)==0x7c00 && (hi&0x3ff);
uint32_t result=geometry;
if(geometry==UINT32_C(0xfbff7bff) || (lo_nan && hi_nan))
{
const float* scene=(const float*)p.scene;
const float* matrix=(const float*)p.matrix;
const float px=(bx+.5f)*((float)p.render_width/p.width);
const float py=(by+.5f)*((float)p.render_height/p.height);
const float cx=d4r_native_project(matrix,0,px,py,best),cy=d4r_native_project(matrix,1,px,py,best);
const float cw=d4r_native_project(matrix,3,px,py,best);
const float reciprocal=cw!=0.f ? 1.f/cw : 0.f;
const _Float16 mx=(_Float16)(cx*reciprocal-scene[6]*px);
const _Float16 my=(_Float16)(cy*reciprocal-scene[7]*py);
result=(uint32_t)__builtin_bit_cast(uint16_t,mx) | (uint32_t)__builtin_bit_cast(uint16_t,my)<<16;
}
*(uint32_t*)((uint8_t*)p.output+(uint64_t)y*p.output_pitch+4*x)=result;
}
__attribute__((device,always_inline)) static inline void d4r_native_motion_compose_impl(D4rNativeMotionParams p,float jitter_x=0,float jitter_y=0)
{
const uint32_t t=__builtin_amdgcn_workitem_id_x();
const uint32_t x=__builtin_amdgcn_workgroup_id_x()*16+(t&15);
const uint32_t y=__builtin_amdgcn_workgroup_id_y()*16+(t>>4);
if(x>=p.width || y>=p.height) return;
const uint8_t* depth=(const uint8_t*)p.depth;
const int cx=clamp_index((int)__builtin_floorf(x+.5f+jitter_x*p.width/p.render_width),p.width);
const int cy=clamp_index((int)__builtin_floorf(y+.5f+jitter_y*p.height/p.render_height),p.height);
uint32_t bx=cx,by=cy;
float best=*(const float*)(depth+(uint64_t)cy*p.depth_pitch+4*cx);
// Same center-preserving, row-major strict-nearer tie rule as the CPU reference.
for(int dy=-1;dy<=1;dy++) for(int dx=-1;dx<=1;dx++)
{
const int nx=cx+dx,ny=cy+dy;
if(nx<0 || ny<0 || nx>=(int)p.width || ny>=(int)p.height) continue;
const float z=*(const float*)(depth+(uint64_t)ny*p.depth_pitch+4*(uint32_t)nx);
if(z<best) { best=z;bx=nx;by=ny; }
}
d4r_native_motion_finish(p,x,y,bx,by,best);
}
// Thin-silhouette halo: shared depth plus row/column bit masks replace repeated
// foreground run scans. Unusual footprints retain the direct depth fallback.
#define D4R_LDS_SYNC() do{__builtin_amdgcn_fence(__ATOMIC_RELEASE,"workgroup");__builtin_amdgcn_s_barrier();__builtin_amdgcn_fence(__ATOMIC_ACQUIRE,"workgroup");}while(0)
#define D4R_LDS __attribute__((shared))
template<bool Hybrid=false>
__attribute__((device,always_inline)) static inline void d4r_native_motion_compose_thin_impl(D4rNativeMotionParams p,float jitter_x,float jitter_y,uint64_t game_motion=0,uint64_t game_pitch=0)
{
// Align the tile to the jittered grid; halo and run checks reach at most ten pixels.
constexpr int margin=Hybrid ? 11 : 10,span=16+2*margin;
D4R_LDS float tile[span*span];
D4R_LDS uint64_t candidate_row[span]; // one bit per eligible thin foreground texel
D4R_LDS uint64_t foreground_row[span],foreground_column[span];
D4R_LDS uint64_t wave_candidates[Hybrid ? 8 : 1];
const uint32_t t=__builtin_amdgcn_workitem_id_x();
const int x0=(int)__builtin_amdgcn_workgroup_id_x()*16,y0=(int)__builtin_amdgcn_workgroup_id_y()*16;
const int gx0=x0+(int)__builtin_floorf(.5f+jitter_x*p.width/p.render_width)-margin;
const int gy0=y0+(int)__builtin_floorf(.5f+jitter_y*p.height/p.render_height)-margin;
const uint8_t* depth=(const uint8_t*)p.depth;
const float outside=__builtin_inff(); // behaves as a missing neighbour in every comparison below
for(int i=t;i<span*span;i+=256)
{
const int gx=gx0+i%span,gy=gy0+i/span;
const float z=gx>=0 && gy>=0 && gx<(int)p.width && gy<(int)p.height ?
*(const float*)(depth+(uint64_t)gy*p.depth_pitch+4*(uint32_t)gx) : outside;
tile[i]=z;
}
D4R_LDS_SYNC();
const int lane=t&31,wave=t>>5;
// Pack the binary foreground test once in both directions. A foreground
// center is thick on an axis iff its nine-bit neighbourhood contains a
// contiguous run of five (all five possible runs include the center).
for(int row=wave;row<span;row+=8)
{
const unsigned r0=__builtin_amdgcn_ballot_w32(tile[row*span+lane]<.99999f);
const unsigned c0=__builtin_amdgcn_ballot_w32(tile[lane*span+row]<.99999f);
const unsigned r1=__builtin_amdgcn_ballot_w32(lane<span-32 && tile[row*span+32+lane]<.99999f);
const unsigned c1=__builtin_amdgcn_ballot_w32(lane<span-32 && tile[(32+lane)*span+row]<.99999f);
if(lane==0)
{
foreground_row[row]=(uint64_t)r0 | (uint64_t)r1<<32;
foreground_column[row]=(uint64_t)c0 | (uint64_t)c1<<32;
}
}
D4R_LDS_SYNC();
uint64_t nearby_candidates=0;
for(int row=wave;row<span;row+=8)
{
uint64_t bits=0;
for(int part=0;part<2;part++)
{
const int col=lane+32*part;
bool thin=false;
if(col>=4 && col<span-4 && row>=4 && row<span-4 && (foreground_row[row]>>col&1))
{
const unsigned horizontal=(foreground_row[row]>>(col-4))&511;
const unsigned vertical=(foreground_column[col]>>(row-4))&511;
const bool thick_x=(horizontal & (horizontal>>1) & (horizontal>>2) & (horizontal>>3) & (horizontal>>4))!=0;
const bool thick_y=(vertical & (vertical>>1) & (vertical>>2) & (vertical>>3) & (vertical>>4))!=0;
thin=!thick_x || !thick_y;
}
const unsigned mask=__builtin_amdgcn_ballot_w32(thin);
bits|=(uint64_t)mask<<(32*part);
}
if(lane==0) candidate_row[row]=bits;
if constexpr(Hybrid) nearby_candidates|=bits;
}
if constexpr(Hybrid) if(lane==0) wave_candidates[wave]=nearby_candidates;
D4R_LDS_SYNC();
const uint32_t x=x0+(t&15),y=y0+(t>>4);
if(x>=p.width || y>=p.height) return;
auto depth_at=[&](int nx,int ny)->float
{
const int lx=nx-gx0,ly=ny-gy0;
if((unsigned)lx<(unsigned)span && (unsigned)ly<(unsigned)span) return tile[ly*span+lx];
if(nx<0 || ny<0 || nx>=(int)p.width || ny>=(int)p.height) return outside;
return *(const float*)(depth+(uint64_t)ny*p.depth_pitch+4*(uint32_t)nx);
};
const int cx=clamp_index((int)__builtin_floorf(x+.5f+jitter_x*p.width/p.render_width),p.width);
const int cy=clamp_index((int)__builtin_floorf(y+.5f+jitter_y*p.height/p.render_height),p.height);
if constexpr(Hybrid)
{
uint64_t any=0;for(int i=0;i<8;i++) any|=wave_candidates[i];
bool keep_native=false;
const bool roi_fits=cx-gx0>=10 && cx-gx0<span-10 && cy-gy0>=10 && cy-gy0<span-10;
if(!roi_fits) keep_native=true; // unusual clamped footprints keep the complete native path
else if(any)
{
const uint64_t window=UINT64_C(8191)<<(cx-gx0-6);
for(int dy=-6;dy<=6;dy++) keep_native|=(candidate_row[cy+dy-gy0]&window)!=0;
}
if(!keep_native)
{
const unsigned mx=(unsigned)((x+.5f)*p.render_width/p.width);
const unsigned my=(unsigned)((y+.5f)*p.render_height/p.height);
*(uint32_t*)((uint8_t*)p.output+(uint64_t)y*p.output_pitch+4*x)=
*(const uint32_t*)(game_motion+(uint64_t)my*game_pitch+4*mx);
return;
}
}
uint32_t bx=cx,by=cy;
const float centre=depth_at(cx,cy);
float best=centre;
for(int dy=-1;dy<=1;dy++) for(int dx=-1;dx<=1;dx++)
{
const int nx=cx+dx,ny=cy+dy;
if(nx<0 || ny<0 || nx>=(int)p.width || ny>=(int)p.height) continue;
const float z=depth_at(nx,ny);
if(z<best) { best=z;bx=nx;by=ny; }
}
// Cover the render filter footprint only around thin silhouettes against
// sky. Long runs on both axes belong to thick surfaces and stay excluded.
const bool tile_reaches=cx-gx0>=10 && cx-gx0<span-10 && cy-gy0>=10 && cy-gy0<span-10;
if(centre>=.99999f)
{
const int sx=(p.width+p.render_width/2)/p.render_width;
const int sy=(p.height+p.render_height/2)/p.render_height;
const int radius=2*(sx>sy ? sx : sy)>6 ? 6 : 2*(sx>sy ? sx : sy);
if(tile_reaches)
{
// Row-major, strictly nearer wins, exactly the scan below but visiting only candidate pixels.
const int left=cx-radius-gx0;
const uint64_t window=((uint64_t)1<<(2*radius+1))-1;
for(int dy=-radius;dy<=radius;dy++)
{
const int row=cy+dy-gy0;
uint64_t m=(candidate_row[row]>>left)&window;
if(dy>=-1 && dy<=1) m&=~(((uint64_t)7)<<(radius-1));
while(m)
{
const int bit=__builtin_ctzll(m);
m&=m-1;
const float z=tile[row*span+left+bit];
if(z<best) { best=z;bx=cx-radius+bit;by=cy+dy; }
}
}
}
else
for(int dy=-radius;dy<=radius;dy++) for(int dx=-radius;dx<=radius;dx++)
{
if(dx>=-1 && dx<=1 && dy>=-1 && dy<=1) continue;
const int nx=cx+dx,ny=cy+dy;
if(nx<0 || ny<0 || nx>=(int)p.width || ny>=(int)p.height) continue;
const float z=depth_at(nx,ny);
if(!(z<best && z<.99999f)) continue;
bool thin=false;
for(int axis=0;axis<2 && !thin;axis++)
{
int run=1;
for(int sign=-1;sign<=1;sign+=2)
for(int step=1;step<5 && run<5;step++)
{
const int tx=nx+(axis==0 ? sign*step : 0),ty=ny+(axis==1 ? sign*step : 0);
if(tx<0 || ty<0 || tx>=(int)p.width || ty>=(int)p.height ||
!(depth_at(tx,ty)<.99999f)) break;
run++;
}
thin=run<5;
}
if(thin) { best=z;bx=nx;by=ny; }
}
}
d4r_native_motion_finish(p,x,y,bx,by,best);
}
extern "C" __attribute__((global)) void d4r_native_motion_compose(D4rNativeMotionParams p)
{ d4r_native_motion_compose_impl(p); }
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256, 256))) void d4r_native_motion_compose_thin(D4rNativeMotionThinParams p)
{ d4r_native_motion_compose_thin_impl(p.base,p.jitter_x,p.jitter_y); }
// Alignment without the extra silhouette halo; wire-color repair is independent.
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256,256))) void d4r_native_motion_compose_aligned(D4rNativeMotionThinParams p)
{ d4r_native_motion_compose_impl(p.base,p.jitter_x,p.jitter_y); }
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256,256))) void d4r_native_motion_compose_hybrid(D4rNativeMotionHybridParams p)
{ d4r_native_motion_compose_thin_impl<true>(p.native.base,p.native.jitter_x,p.native.jitter_y,p.game_motion,p.game_motion_pitch); }
kernels/support/native_motion_prepared.hip
+133
View File
@@ -0,0 +1,133 @@
// Exact native composition with shared wire geometry; wave64 compose, wave32 preparation.
#pragma clang fp contract(off)
#define NMP_SYNC() do { __builtin_amdgcn_fence(__ATOMIC_RELEASE,"workgroup"); __builtin_amdgcn_s_barrier(); __builtin_amdgcn_fence(__ATOMIC_ACQUIRE,"workgroup"); } while(0)
__attribute__((device,always_inline,target("wavefrontsize64"))) static inline float d4r_nmp_project64(const float* m,int c,float x,float y,float z)
{
float value=m[c]*x;
value=__builtin_fmaf(m[4+c],y,value);
value=__builtin_fmaf(m[8+c],z,value);
return value+m[12+c];
}
__attribute__((device,always_inline,target("wavefrontsize64"))) static inline void d4r_nmp_finish64(const D4rNativeMotionParams& p,
uint32_t x,uint32_t y,uint32_t bx,uint32_t by,float best)
{
const uint32_t geometry=*(const uint32_t*)((const uint8_t*)p.geometry+(uint64_t)by*p.geometry_pitch+4*bx);
const uint16_t lo=geometry,hi=geometry>>16;
const bool lo_nan=(lo&0x7c00)==0x7c00 && (lo&0x3ff);
const bool hi_nan=(hi&0x7c00)==0x7c00 && (hi&0x3ff);
uint32_t result=geometry;
if(geometry==UINT32_C(0xfbff7bff) || (lo_nan && hi_nan))
{
const float* scene=(const float*)p.scene;
const float* matrix=(const float*)p.matrix;
const float px=(bx+.5f)*((float)p.render_width/p.width);
const float py=(by+.5f)*((float)p.render_height/p.height);
const float cx=d4r_nmp_project64(matrix,0,px,py,best),cy=d4r_nmp_project64(matrix,1,px,py,best);
const float cw=d4r_nmp_project64(matrix,3,px,py,best);
const float reciprocal=cw!=0.f ? 1.f/cw : 0.f;
const _Float16 mx=(_Float16)(cx*reciprocal-scene[6]*px);
const _Float16 my=(_Float16)(cy*reciprocal-scene[7]*py);
result=(uint32_t)__builtin_bit_cast(uint16_t,mx) | (uint32_t)__builtin_bit_cast(uint16_t,my)<<16;
}
*(uint32_t*)((uint8_t*)p.output+(uint64_t)y*p.output_pitch+4*x)=result;
}
__attribute__((device,always_inline,target("wavefrontsize64"))) static inline unsigned d4r_nmp_bits64(const unsigned* data,int words,int left)
{
int word=left>=0 ? left/32 : -1,shift=left&31;
unsigned lo=word>=0 && word<words ? data[word] : 0;
unsigned hi=word+1>=0 && word+1<words ? data[word+1] : 0;
return (unsigned)(((uint64_t)lo | (uint64_t)hi<<32)>>shift);
}
__attribute__((device,always_inline,target("wavefrontsize64"))) static inline int d4r_nmp_clamp64(int x,unsigned n) {return x<0 ? 0 : x>=(int)n ? (int)n-1 : x;}
extern "C" __attribute__((global,amdgpu_flat_work_group_size(256,256),target("wavefrontsize64"))) void d4r_native_motion_prepared(D4rNativeMotionPreparedParams a)
{
const auto& p=a.native.base;
unsigned t=__builtin_amdgcn_workitem_id_x();
int x0=__builtin_amdgcn_workgroup_id_x()*16,y0=__builtin_amdgcn_workgroup_id_y()*16;
int left=x0+(int)__builtin_floorf(.5f+a.native.jitter_x*p.width/p.render_width)-8;
int top=y0+(int)__builtin_floorf(.5f+a.native.jitter_y*p.height/p.render_height)-8;
const int rw=(p.width+31)/32;
__attribute__((shared)) unsigned rows[32],any;
if(t<32) {
int y=top+t;unsigned bits=y>=0 && y<(int)p.height ? d4r_nmp_bits64((const unsigned*)a.thin_mask+(uint64_t)y*rw,rw,left) : 0;
rows[t]=bits;unsigned nonempty=(unsigned)__builtin_amdgcn_ballot_w64(bits!=0);if(t==0) any=nonempty;
}
NMP_SYNC();
unsigned x=x0+(t&15),y=y0+(t>>4);
if(x>=p.width || y>=p.height) return;
const int cx=d4r_nmp_clamp64((int)__builtin_floorf(x+.5f+a.native.jitter_x*p.width/p.render_width),p.width);
const int cy=d4r_nmp_clamp64((int)__builtin_floorf(y+.5f+a.native.jitter_y*p.height/p.render_height),p.height);
const uint8_t* depth=(const uint8_t*)p.depth;
const float center=*(const float*)(depth+(uint64_t)cy*p.depth_pitch+4*cx);
float best=center;unsigned bx=cx,by=cy;
for(int dy=-1;dy<=1;dy++)for(int dx=-1;dx<=1;dx++) {
int nx=cx+dx,ny=cy+dy;if(nx<0 || ny<0 || nx>=(int)p.width || ny>=(int)p.height)continue;
float z=*(const float*)(depth+(uint64_t)ny*p.depth_pitch+4*nx);
if(z<best){best=z;bx=nx;by=ny;}
}
if(center>=.99999f) {
int sx=(p.width+p.render_width/2)/p.render_width,sy=(p.height+p.render_height/2)/p.render_height;
int radius=2*(sx>sy ? sx : sy);if(radius>6)radius=6;
bool cached=cx-radius>=left && cx+radius<left+32 && cy-radius>=top && cy+radius<top+32;
if(any || !cached)for(int dy=-radius;dy<=radius;dy++) {
int ny=cy+dy;if(ny<0 || ny>=(int)p.height)continue;
unsigned m=(cached ? rows[ny-top]>>(cx-radius-left) : d4r_nmp_bits64((const unsigned*)a.thin_mask+(uint64_t)ny*rw,rw,cx-radius)) & ((1u<<(2*radius+1))-1);
if(dy>=-1 && dy<=1)m&=~(7u<<(radius-1));
while(m) {int bit=__builtin_ctz(m);m&=m-1;int nx=cx-radius+bit;
float z=*(const float*)(depth+(uint64_t)ny*p.depth_pitch+4*nx);
if(z<best){best=z;bx=nx;by=ny;}
}
}
}
d4r_nmp_finish64(p,x,y,bx,by,best);
}
extern "C" __attribute__((global,amdgpu_flat_work_group_size(256,256))) void d4r_native_motion_prepare(D4rNativeMotionPreparedParams a)
{
const auto& p=a.native.base;
__attribute__((shared)) float z[40][41];
__attribute__((shared)) uint64_t row_bits[40],col_bits[40];
unsigned t=__builtin_amdgcn_workitem_id_x(),lane=t&31,wave=t>>5;
const int x0=(int)__builtin_amdgcn_workgroup_id_x()*32,y0=(int)__builtin_amdgcn_workgroup_id_y()*32;
for(int i=t;i<1600;i+=256) {
int ly=i/40,lx=i%40,gx=clamp_index(x0+lx-4,p.width),gy=clamp_index(y0+ly-4,p.height);
z[ly][lx]=*(const float*)(p.depth+(uint64_t)gy*p.depth_pitch+4*gx);
}
NMP_SYNC();
for(int r=wave;r<40;r+=8) {
uint64_t rb=0,cb=0;
for(int part=0;part<2;part++) {
int c=lane+32*part;
bool rf=c<40 && x0+c-4>=0 && x0+c-4<(int)p.width && y0+r-4>=0 && y0+r-4<(int)p.height && z[r][c]<.99999f;
bool cf=c<40 && y0+c-4>=0 && y0+c-4<(int)p.height && x0+r-4>=0 && x0+r-4<(int)p.width && z[c][r]<.99999f;
rb|=(uint64_t)__builtin_amdgcn_ballot_w32(rf)<<(part*32);cb|=(uint64_t)__builtin_amdgcn_ballot_w32(cf)<<(part*32);
}
if(lane==0){row_bits[r]=rb;col_bits[r]=cb;}
}
NMP_SYNC();
D4rWireHistoryParams wp={};wp.width=p.width;wp.height=p.height;wp.native_depth=p.depth;wp.depth_pitch=p.depth_pitch;wp.reserved=a.wire_flags;
auto depth=[&](int gx,int gy)->float {return z[gy-y0+4][gx-x0+4];};
for(int r=wave;r<32;r+=8) {
const int gx=x0+lane,gy=y0+r,lx=lane+4,ly=r+4;
unsigned h=(row_bits[ly]>>lane)&511,v=(col_bits[lx]>>r)&511;
bool hx=(h&(h>>1)&(h>>2)&(h>>3)&(h>>4))!=0,hy=(v&(v>>1)&(v>>2)&(v>>3)&(v>>4))!=0;
bool thin=(h&16)!=0 && (!hx || !hy);
unsigned bits=__builtin_amdgcn_ballot_w32(thin);
if(lane==0 && gy<(int)p.height) ((unsigned*)a.thin_mask)[(uint64_t)gy*((p.width+31)/32)+x0/32]=bits;
if(a.wire_axes && gx<(int)p.width && gy<(int)p.height) {
// Existing foreground bits reject sky centers and non-sky endpoints
// without repeating LDS depth reads. The reference still handles NaNs,
// edge clamping and relative-depth separation for every surviving axis.
unsigned axes=0;
if(h&16) {
const bool sky_only=(a.wire_flags & D4R_WIRE_HISTORY_SKY_ONLY)!=0;
if(!sky_only || !(h&257)) axes|=wh_thin_interior(wp,gx,gy,0,depth)?1u:0u;
if(!sky_only || !(v&257)) axes|=wh_thin_interior(wp,gx,gy,1,depth)?2u:0u;
}
((uint8_t*)a.wire_axes)[(uint64_t)gy*p.width+gx]=axes;
}
}
}
#undef NMP_SYNC
kernels/support/wire_history.hip
+248
View File
@@ -0,0 +1,248 @@
// Geometry-confirmed short-lived color history for locally separated thin silhouettes.
#include "../../tools/d4r_wire_history.h"
#pragma clang fp contract(off)
#define WH_SYNC() do { __builtin_amdgcn_fence(__ATOMIC_RELEASE,"workgroup"); __builtin_amdgcn_s_barrier(); __builtin_amdgcn_fence(__ATOMIC_ACQUIRE,"workgroup"); } while(0)
__attribute__((device,always_inline)) static inline int wh_clamp(int v,int lo,int hi)
{ return v<lo ? lo : v>hi ? hi : v; }
__attribute__((device,always_inline)) static inline float wh_depth(const D4rWireHistoryParams& p,int x,int y)
{ x=wh_clamp(x,0,p.width-1);y=wh_clamp(y,0,p.height-1);return *(const float*)(p.native_depth+(uint64_t)y*p.depth_pitch+4*x); }
template<class Depth>
__attribute__((device,always_inline)) static inline bool wh_thin_axis(const D4rWireHistoryParams& p,int x,int y,int axis,Depth depth)
{
float center=depth(x,y);if(!(center>=0 && center<.99999f)) return false;
if((p.reserved & D4R_WIRE_HISTORY_SKY_ONLY) &&
(!(depth(x-4*(axis==0),y-4*(axis==1))>=.99999f) || !(depth(x+4*(axis==0),y+4*(axis==1))>=.99999f))) return false;
float epsilon=.25f*(1-center);if(epsilon<2e-6f) epsilon=2e-6f;
float limit=center+epsilon;
int run=1;
for(int sign=-1;sign<=1;sign+=2) for(int step=1;step<5 && run<5;step++)
{
int tx=x+(axis==0 ? sign*step : 0),ty=y+(axis==1 ? sign*step : 0);
if(tx<0 || ty<0 || tx>=(int)p.width || ty>=(int)p.height || !(depth(tx,ty)<limit)) break;
run++;
}
if(run>=5) return false;
float gap=.5f*(1-center);if(gap<2e-6f) gap=2e-6f;
bool sides[2]={false,false};
for(int sign=-1;sign<=1;sign+=2) for(int step=1;step<=4;step++)
sides[(sign+1)/2]|=depth(x+sign*step*(axis==0),y+sign*step*(axis==1))>center+gap;
return sides[0] && sides[1];
}
// Interior specialization: fixed eight depth loads, bit scans for the first
// non-near neighbour on each side, and the original strict separation test.
// Edge samples keep the bounds-aware reference above.
template<class Depth>
__attribute__((device,always_inline)) static inline bool wh_thin_interior(const D4rWireHistoryParams& p,int x,int y,int axis,Depth depth)
{
if(x<4 || y<4 || x>=(int)p.width-4 || y>=(int)p.height-4)
return wh_thin_axis(p,x,y,axis,depth);
const float center=depth(x,y);
if(!(center>=0 && center<.99999f)) return false;
if((p.reserved & D4R_WIRE_HISTORY_SKY_ONLY) &&
(!(depth(x-4*(axis==0),y-4*(axis==1))>=.99999f) || !(depth(x+4*(axis==0),y+4*(axis==1))>=.99999f))) return false;
float epsilon=.25f*(1-center);if(epsilon<2e-6f) epsilon=2e-6f;
float gap=.5f*(1-center);if(gap<2e-6f) gap=2e-6f;
const float limit=center+epsilon,far_limit=center+gap;
unsigned negative=0,positive=0;
bool far_negative=false,far_positive=false;
#pragma unroll
for(int step=1;step<=4;step++)
{
const float n=depth(x-(axis==0 ? step : 0),y-(axis==1 ? step : 0));
negative|=(unsigned)(!(n<limit))<<(step-1);
far_negative|=n>far_limit;
}
// Four contiguous near neighbours already prove a run of at least five.
// No positive-side depths or separation checks can make it thin.
if(!negative) return false;
#pragma unroll
for(int step=1;step<=4;step++)
{
const float v=depth(x+(axis==0 ? step : 0),y+(axis==1 ? step : 0));
positive|=(unsigned)(!(v<limit))<<(step-1);
far_positive|=v>far_limit;
}
if(!positive) return false;
return __builtin_ctz(negative|16)+__builtin_ctz(positive|16)<4 && far_negative && far_positive;
}
__attribute__((device,always_inline)) static inline float wh_luma(uint64_t rgb,uint64_t pitch,int x,int y)
{
const _Float16* c=(const _Float16*)(rgb+(uint64_t)y*pitch+8*x);
float l=(float)c[0]*.2126f+(float)c[1]*.7152f+(float)c[2]*.0722f;
return l>1e-5f ? l : 1e-5f;
}
// Median in log space is the geometric mean of the two central luminances.
__attribute__((device,always_inline)) static inline float wh_background(uint64_t rgb,uint64_t pitch,int x,int y,int ax,int ay,bool input)
{
float v[6];int count=input ? 4 : 6;
const int offsets[6]={-12,-10,-8,8,10,12};
const int small[4]={-5,-4,4,5};
for(int i=0;i<count;i++) { int d=input ? small[i] : offsets[i];v[i]=wh_luma(rgb,pitch,x+d*ax,y+d*ay); }
for(int i=1;i<count;i++) for(int j=i;j>0 && v[j]<v[j-1];j--) {float a=v[j];v[j]=v[j-1];v[j-1]=a;}
return __builtin_sqrtf(v[count/2-1]*v[count/2]);
}
template<bool Sparse,bool CachedAxes=false>
__attribute__((device,always_inline)) static inline void wh_run(D4rWireHistoryParams p,uint64_t patch_mask,uint64_t prepared_axes=0)
{
uint32_t t=__builtin_amdgcn_workitem_id_x();
int x=__builtin_amdgcn_workgroup_id_x()*16+(t&15),y=__builtin_amdgcn_workgroup_id_y()*16+(t>>4);
// Classify each nearby depth texel once, shared by the four bilinear samples
// of adjacent output pixels. All lanes reach both barriers, including partial tiles.
constexpr int span=26,margin=5;
__attribute__((shared)) float tile[span*span];
__attribute__((shared)) uint8_t axes[span*span];
const int ox=(int)__builtin_amdgcn_workgroup_id_x()*16+(int)__builtin_floorf(p.jitter_x*p.width/p.render_width)-margin;
const int oy=(int)__builtin_amdgcn_workgroup_id_y()*16+(int)__builtin_floorf(p.jitter_y*p.height/p.render_height)-margin;
if constexpr(!CachedAxes) {
for(int i=t;i<span*span;i+=256) tile[i]=wh_depth(p,ox+i%span,oy+i/span);
WH_SYNC();
}
auto depth=[&](int tx,int ty)->float {
int lx=tx-ox,ly=ty-oy;
return (unsigned)lx<span && (unsigned)ly<span ? tile[ly*span+lx] : wh_depth(p,tx,ty);
};
auto tile_depth=[&](int tx,int ty)->float { return tile[(ty-oy)*span+tx-ox]; };
if constexpr(!CachedAxes) {
for(int i=t;i<span*span;i+=256)
{
int lx=i%span,ly=i/span;
uint8_t mask=0;
if(lx>=4 && lx<span-4 && ly>=4 && ly<span-4)
{
if(wh_thin_interior(p,ox+lx,oy+ly,0,tile_depth)) mask|=1;
if(wh_thin_interior(p,ox+lx,oy+ly,1,tile_depth)) mask|=2;
}
axes[i]=mask;
}
WH_SYNC();
}
auto thin_axes=[&](int tx,int ty)->uint8_t {
if constexpr(CachedAxes) return *(const uint8_t*)(prepared_axes+(uint64_t)ty*p.width+tx);
else {
int lx=tx-ox,ly=ty-oy;
if(lx>=4 && lx<span-4 && ly>=4 && ly<span-4) return axes[ly*span+lx];
return (wh_thin_axis(p,tx,ty,0,depth) ? 1 : 0) | (wh_thin_axis(p,tx,ty,1,depth) ? 2 : 0);
}
};
if(x>=(int)p.width || y>=(int)p.height) return;
uint64_t offset=(uint64_t)y*p.output_pitch+8*x,index=(uint64_t)y*p.width+x;
if constexpr(Sparse) *(uint8_t*)(patch_mask+index)=0;
else *(uint64_t*)(p.output+offset)=*(const uint64_t*)(p.current+offset);
// Age 255 makes this entry unreadable by the next frame. The prepared
// path need not copy raw RGB or initialize depth for an unobserved entry.
if constexpr(!CachedAxes) *(uint64_t*)(p.output_history+offset)=*(const uint64_t*)(p.current+offset);
*(uint8_t*)(p.output_age+index)=255;
if constexpr(!CachedAxes) *(float*)(p.output_depth+4*index)=1.f;
if(x<16 || y<16 || x>=(int)p.width-16 || y>=(int)p.height-16) return;
float nx=x+p.jitter_x*p.width/p.render_width,ny=y+p.jitter_y*p.height/p.render_height;
int fx=(int)__builtin_floorf(nx),fy=(int)__builtin_floorf(ny),sx=0,sy=0;
float coverage=0,best_weight=-1;
for(int dy=0;dy<=1;dy++) for(int dx=0;dx<=1;dx++)
{
int tx=wh_clamp(fx+dx,0,p.width-1),ty=wh_clamp(fy+dy,0,p.height-1);
if(!thin_axes(tx,ty)) continue;
float weight=(1-__builtin_fabsf(nx-tx))*(1-__builtin_fabsf(ny-ty));coverage+=weight;
if(weight>best_weight) { best_weight=weight;sx=tx;sy=ty; }
}
if(coverage<=.05f) return;
int ay=(thin_axes(sx,sy)&2) ? 1 : 0,ax=1-ay;
const bool conservative=(p.reserved & D4R_WIRE_HISTORY_CONSERVATIVE)!=0;
const int radius=conservative ? 1 : 2,age_limit=conservative ? 4 : 16;
float current_z=wh_depth(p,sx,sy);
float tolerance=(conservative ? .01f : .02f)*(1-current_z);
const float depth_floor=conservative ? 2e-7f : 2e-6f;
if(tolerance<depth_floor) tolerance=depth_floor;
if(p.reserved & D4R_WIRE_HISTORY_SKY_ONLY)
{
// Fractional wire coverage may overlap unrelated foreground geometry.
// Preserve that surface rather than dragging wire color onto it. Sky
// and depth-matched wire centers remain eligible; NaN depth is rejected.
const float nearest_z=wh_depth(p,(int)__builtin_floorf(nx+.5f),(int)__builtin_floorf(ny+.5f));
if(!(nearest_z>=.99999f) && !(__builtin_fabsf(nearest_z-current_z)<tolerance)) return;
}
*(float*)(p.output_depth+4*index)=current_z;
int ix=wh_clamp((int)__builtin_floorf((x+.5f)*p.render_width/p.width+p.jitter_x),5,p.render_width-6);
int iy=wh_clamp((int)__builtin_floorf((y+.5f)*p.render_height/p.height+p.jitter_y),5,p.render_height-6);
float source_x=(x+.5f)*p.render_width/p.width+p.jitter_x;
float source_y=(y+.5f)*p.render_height/p.height+p.jitter_y;
float best_distance=1e30f;int source_ix=ix,source_iy=iy;
for(int dy=-radius;dy<=radius;dy++) for(int dx=-radius;dx<=radius;dx++)
{
int tx=ix+dx,ty=iy+dy;
float value=*(const float*)(p.input_depth+(uint64_t)ty*p.input_depth_pitch+4*tx);
if(!(__builtin_fabsf(value-current_z)<tolerance)) continue;
float xx=tx+.5f-source_x,yy=ty+.5f-source_y,distance=xx*xx+yy*yy;
if(distance<best_distance) {best_distance=distance;source_ix=tx;source_iy=ty;}
}
uint64_t target=0;
if(best_distance!=1e30f)
{
target=p.color+(uint64_t)source_iy*p.color_pitch+8*source_ix;
*(uint64_t*)(p.output_history+offset)=*(const uint64_t*)target;
*(uint8_t*)(p.output_age+index)=0;
}
else if(p.history_valid)
{
const _Float16* motion=(const _Float16*)(p.motion+(uint64_t)y*p.motion_pitch+4*x);
float px=x+(float)motion[0]*p.width,py=y+(float)motion[1]*p.height;
if(!(px>=16 && py>=16 && px<p.width-16 && py<p.height-16)) return;
float best=1e30f;int hx=0,hy=0;
for(int dy=-radius;dy<=radius;dy++) for(int dx=-radius;dx<=radius;dx++)
{
int tx=wh_clamp((int)__builtin_rintf(px+dx),16,p.width-17),ty=wh_clamp((int)__builtin_rintf(py+dy),16,p.height-17);
uint64_t previous_index=(uint64_t)ty*p.width+tx;
if(*(const uint8_t*)(p.previous_age+previous_index)>=age_limit) continue;
float previous_z=*(const float*)(p.previous_depth+4*previous_index),limit=(conservative ? .01f : .05f)*(1-current_z);if(limit<depth_floor) limit=depth_floor;
if(!(__builtin_fabsf(previous_z-current_z)<limit)) continue;
// Brightness is not correspondence: selecting the darkest neighbour can
// drag a previous wire sample sideways along a camera pan.
const float ddx=tx-px,ddy=ty-py;
float value=conservative ? ddx*ddx+ddy*ddy : wh_luma(p.previous,p.output_pitch,tx,ty);
if(value<best) {best=value;hx=tx;hy=ty;}
}
if(best!=1e30f)
{
uint64_t hi=(uint64_t)hy*p.width+hx;
target=p.previous+(uint64_t)hy*p.output_pitch+8*hx;
*(uint64_t*)(p.output_history+offset)=*(const uint64_t*)target;
*(uint8_t*)(p.output_age+index)=*(const uint8_t*)(p.previous_age+hi)+1;
}
}
if(!target) return;
float background=wh_background(p.current,p.output_pitch,x,y,ax,ay,false);
float current=wh_luma(p.current,p.output_pitch,x,y);
float value=wh_luma(target,8,0,0),ratio=value/background;
float threshold=background*__builtin_sqrtf(ratio*__builtin_sqrtf(ratio));
if(!((ratio<.8408964153f && current>threshold) || (ratio>1.189207115f && current<threshold))) return;
if constexpr(Sparse)
{
*(uint64_t*)(p.output+offset)=*(const uint64_t*)(p.current+offset);
*(uint8_t*)(patch_mask+index)=1;
}
if(coverage>1) coverage=1;
const _Float16* observed=(const _Float16*)target;
const _Float16* now=(_Float16*)(p.current+offset);_Float16* out=(_Float16*)(p.output+offset);
for(int c=0;c<3;c++)
{
float result=(float)now[c]*(1-coverage)+(float)observed[c]*coverage;
if(result<0) result=0;if(result>65504) result=65504;out[c]=(_Float16)result;
}
}
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256,256))) void d4r_wire_history(D4rWireHistoryParams p)
{ wh_run<false>(p,0); }
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256,256))) void d4r_wire_history_sparse(D4rWireHistorySparseParams p)
{ wh_run<true>(p.base,p.patch_mask); }
extern "C" __attribute__((global, amdgpu_flat_work_group_size(256,256))) void d4r_wire_history_apply(D4rWireHistorySparseParams p)
{
const unsigned t=__builtin_amdgcn_workitem_id_x();
const unsigned x=__builtin_amdgcn_workgroup_id_x()*16+(t&15),y=__builtin_amdgcn_workgroup_id_y()*16+(t>>4);
if(x>=p.base.width || y>=p.base.height || !*(const uint8_t*)(p.patch_mask+(uint64_t)y*p.base.width+x)) return;
const uint64_t offset=(uint64_t)y*p.base.output_pitch+8*x;
*(uint64_t*)(p.base.current+offset)=*(const uint64_t*)(p.base.output+offset);
}
#undef WH_SYNC
extern "C" __attribute__((global,amdgpu_flat_work_group_size(256,256))) void d4r_wire_history_preclassified(D4rWireHistoryPreparedParams p)
{ wh_run<true,true>(p.sparse.base,p.sparse.patch_mask,p.axes); }
packaging/d4r.ini
+20 -1
View File
@@ -32,6 +32,18 @@ Enable = auto
; ([Engine]) F runs from the model in d4r\engine\rr-f; frames the library serves instead show
; preset D when the DLL is older (d4r_nvngx.log says so).
Model = auto
; dlss: NVIDIA's DLSS Ray Reconstruction (the default). fsr: experimental AMD FSR Ray Regeneration 1.2,
; then the Super Sampling above upscales. Needs AMD's amd_fidelityfx_denoiser_dx12.dll in the d4r folder
; (or FsrDenoiser = its path). Restart the game after changing it.
Backend = dlss
FsrDenoiser = auto
; Backend = fsr tuning. FsrStabilityBias: the AMD denoiser's temporal stability bias; auto = AMD's
; default (1), which blurs reflections and fine detail; 0 measured sharper and closer to the noise-free
; truth, moving or still (D4R_FSR_RR_STABILITY_BIAS). FsrDetail: the share (0-1) of the colour that goes
; around the denoiser, keeping detail it removes as noise (hair, water ripples, foliage) at the cost of some
; grain; 0 = off (D4R_FSR_RR_DETAIL).
FsrStabilityBias = auto
FsrDetail = 0
[Latency]
; 0: every frame shows its own DLSS result, as with a native upscaler (needs the d3d12.dll that came
@@ -62,6 +74,10 @@ NativeFp8 = on
; Requires render- or display-resolution, unjittered motion vectors and same-frame linear VRAM inputs. Restart to change.
; Unsupported features keep original vectors; larger radii may affect disocclusion edges.
MotionVectorDilation = 0
; Opt-in sky-fill experiment for equal render/motion grids: borrow only from foreground
; runs <=3 pixels along either axis. Opaque dilation is unchanged. Restart to change.
; Unsupported routes retain nearest-foreground fill. (D4R_MOTION_THIN_SKY)
MotionDilationThinSky = false
; Keep inputs and output in video memory. Readiness uses adaptive CPU polling by default.
VramInterop = true
; auto: adaptive input-marker polling; N: fixed interval in microseconds (0 yields).
@@ -96,8 +112,11 @@ CacheDir = auto
[Debug]
; d4r's log is d4r\d4r_nvngx.log, rewritten at every launch.
; Profile = true adds one line of timings per frame.
; Profile = true measures stage timings and writes sampled timing lines.
Profile = false
; Log the first 5 frames, then every Nth frame. 1 = every frame. Restart after changing.
; FSR RR stage summaries average N completed GPU samples. Env: D4R_PROFILE_LOG_EVERY
ProfileLogEvery = 60
[Env]
; Extra environment variables for d4r, NAME = value.
scripts/build_d4r_nvngx_shim.sh
+1 -1
View File
@@ -25,7 +25,7 @@ PARAM_OBJECT="$ROOT/build/d4r_ngx_param_msvc.obj"
"$MINGW_CXX" -std=c++20 -O2 -Wall -Wextra -Wno-missing-field-initializers -DD4R_ENGINE_VK_TABLE -I"$ROOT/build/vulkan-include" -I"$ROOT/build/vulkan-staging" -shared -static -static-libgcc -static-libstdc++ \
"$ROOT/tools/d4r_nvngx_shim.cpp" "$ROOT/tools/d4r_ngx_param_host.cpp" "$PARAM_OBJECT" \
"$ROOT/engine/runtime.cpp" "$ROOT/engine/game.cpp" "$ROOT/engine/game_m.cpp" "$ROOT/engine/game_rr.cpp" "$ROOT/engine/vk_dispatch.cpp" \
-Wl,--entry,d4r_dll_entry -o "$ROOT/build/d4r_nvngx.dll"
-ldxgi -lbcrypt -Wl,--entry,d4r_dll_entry -o "$ROOT/build/d4r_nvngx.dll"
printf 'Built %s\n' "$ROOT/build/d4r_nvngx.dll"
cp "$ROOT/build/d4r_nvngx.dll" "$ROOT/build/d4r.asi"
printf 'Built %s\n' "$ROOT/build/d4r.asi"
scripts/build_fsr_rr_probe.sh
Executable
+15
View File
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
set -euo pipefail
# Builds build/fsr_rr_probe.exe, the standalone AMD FSR Ray Regeneration provider/execution check
# (tools/fsr_rr_probe.cpp). It needs no NVIDIA or d4r runtime files.
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
MINGW_CXX="${MINGW_CXX:-x86_64-w64-mingw32-g++}"
VULKAN_INCLUDE="${VULKAN_INCLUDE:-/usr/include}"
mkdir -p "$ROOT/build/vulkan-include"
ln -sfn "$VULKAN_INCLUDE/vulkan" "$ROOT/build/vulkan-include/vulkan"
ln -sfn "$VULKAN_INCLUDE/vk_video" "$ROOT/build/vulkan-include/vk_video"
"$MINGW_CXX" -std=c++20 -O2 -Wall -Wextra -Wno-missing-field-initializers -municode -static -static-libgcc \
-static-libstdc++ -I"$ROOT/build/vulkan-include" "$ROOT/tools/fsr_rr_probe.cpp" -ld3d12 -ldxgi -o "$ROOT/build/fsr_rr_probe.exe"
printf 'Built %s\n' "$ROOT/build/fsr_rr_probe.exe"
scripts/build_rr_backend_check.sh
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -euo pipefail
# Builds build/rr_backend_check.exe (tools/rr_backend_check.cpp): a DLSS Ray Reconstruction request
# through d4r_nvngx.dll, measured against the synthetic scene's ground truth. Builds the shim first,
# for the NGX parameter object it links against.
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
MINGW_CXX="${MINGW_CXX:-x86_64-w64-mingw32-g++}"
"$ROOT/scripts/build_d4r_nvngx_shim.sh" >/dev/null
"$MINGW_CXX" -std=c++20 -O2 -Wall -Wextra -Wno-missing-field-initializers -static -static-libgcc -static-libstdc++ \
"$ROOT/tools/rr_backend_check.cpp" "$ROOT/tools/d4r_ngx_param_host.cpp" "$ROOT/build/d4r_ngx_param_msvc.obj" \
-ld3d12 -ldxgi -o "$ROOT/build/rr_backend_check.exe"
printf 'Built %s\n' "$ROOT/build/rr_backend_check.exe"
scripts/build_vkd3d_proton_d4r.sh
+17 -2
View File
@@ -6,7 +6,11 @@ set -euo pipefail
# the game launcher takes as D4R_VKD3D_DIR.
# usage: build_vkd3d_proton_d4r.sh OUT_DIR
# Env: VKD3D_PROTON_SRC (clone, default ~/.cache/d4r-vkd3d-proton),
# VKD3D_PROTON_COMMIT (default 3dfc6f07, the base GE-Proton11-3 ships).
# VKD3D_PROTON_COMMIT (default 3dfc6f07, the base GE-Proton11-3 ships),
# D4R_VKD3D_FSR_RR=1: also what AMD's FSR Ray Regeneration needs on RDNA 3
# ([RayReconstruction] Backend = fsr): dxil-spirv's accepted scalar FP8 fallback (upstream PR 307,
# merge f9f6294f, cherry-picked onto the linked translator) and -Denable_extended_emulation=true,
# in a build directory of its own. Run games with DXIL_SPIRV_CONFIG=wmma_rdna3_workaround.
if [[ $# -ne 1 ]]; then
printf 'usage: %s OUT_DIR\n' "$0" >&2
exit 2
@@ -22,7 +26,18 @@ BUILD="$SRC-build"
git -C "$SRC" checkout -q -f -B d4r-split "$COMMIT"
git -C "$SRC" submodule update --init --recursive -q
git -C "$SRC" apply "$ROOT"/patches/vkd3d-proton/*.patch
[[ -d "$BUILD" ]] || meson setup --cross-file "$SRC/build-win64.txt" --buildtype release -Denable_tests=false "$BUILD" "$SRC"
EXTRA=()
if [[ "${D4R_VKD3D_FSR_RR:-0}" == 1 ]]; then
DXIL="$SRC/subprojects/dxil-spirv"
git -C "$DXIL" fetch -q origin f9f6294f3a6795e071bef3a64d2bee99e7c1b7bc
if ! git -C "$DXIL" merge-base --is-ancestor f9f6294f3a6795e071bef3a64d2bee99e7c1b7bc^2 HEAD; then
git -C "$DXIL" -c user.name=d4r -c user.email=d4r@localhost cherry-pick -x \
$(git -C "$DXIL" rev-list --reverse f9f6294f3a6795e071bef3a64d2bee99e7c1b7bc^1..f9f6294f3a6795e071bef3a64d2bee99e7c1b7bc^2)
fi
BUILD="$SRC-build-fsrrr"
EXTRA=(-Denable_extended_emulation=true)
fi
[[ -d "$BUILD" ]] || meson setup --cross-file "$SRC/build-win64.txt" --buildtype release -Denable_tests=false "${EXTRA[@]}" "$BUILD" "$SRC"
ninja -C "$BUILD"
mkdir -p "$OUT"
x86_64-w64-mingw32-strip -o "$OUT/d3d12.dll" "$BUILD/libs/d3d12/d3d12.dll"
scripts/build_vulkan_staging.sh
+3
View File
@@ -5,6 +5,9 @@ OUT="${1:-$ROOT/build/vulkan-staging}"
mkdir -p "$OUT"
glslangValidator -V --target-env vulkan1.0 -o "$OUT/read.spv" "$ROOT/kernels/support/vulkan_read.comp" >/dev/null
glslangValidator -V --target-env vulkan1.0 -o "$OUT/wait.spv" "$ROOT/kernels/support/vulkan_wait.comp" >/dev/null
# FSR Ray Regeneration backend (tools/d4r_fsr_rr_passes.h)
glslangValidator -V --target-env vulkan1.0 -o "$OUT/fsr_rr_prepare.spv" "$ROOT/kernels/support/fsr_rr_prepare.comp" >/dev/null
glslangValidator -V --target-env vulkan1.0 -o "$OUT/fsr_rr_compose.spv" "$ROOT/kernels/support/fsr_rr_compose.comp" >/dev/null
for format in rgba16f rgba32f r11f_g11f_b10f rgb10_a2 rgba8; do
glslangValidator -V --target-env vulkan1.0 -DOUTPUT_FORMAT="$format" \
-o "$OUT/write_$format.spv" "$ROOT/kernels/support/vulkan_write.comp" >/dev/null
scripts/d4r_config.py
+34
View File
@@ -100,6 +100,34 @@ def main(argv):
env["D4R_RR_PRESET"] = rr_model
else:
fail(f"[RayReconstruction] Model {rr_model!r}: use auto, default, D, E or F")
# Backend: which denoiser serves a game's Ray Reconstruction request. dlss (the default) is
# NVIDIA's model (Model above); fsr is AMD's FSR Ray Regeneration, followed by Super Sampling.
rr_backend = get("RayReconstruction", "Backend")
if rr_backend is not None:
if rr_backend.lower() not in ("dlss", "fsr"):
fail(f"[RayReconstruction] Backend {rr_backend!r}: use dlss or fsr")
env["D4R_RR_BACKEND"] = rr_backend.lower()
# FSR denoiser tuning: FsrStabilityBias (auto = AMD's default, or a number >= 0) and FsrDetail (the share,
# 0-1, of the colour that goes around the denoiser; 0 = off).
fsr_stability = get("RayReconstruction", "FsrStabilityBias")
if fsr_stability is not None and fsr_stability.lower() != "auto":
try:
if float(fsr_stability) < 0:
raise ValueError
except ValueError:
fail(f"[RayReconstruction] FsrStabilityBias {fsr_stability!r}: use auto or a number >= 0")
env["D4R_FSR_RR_STABILITY_BIAS"] = fsr_stability
fsr_detail = get("RayReconstruction", "FsrDetail")
if fsr_detail is not None:
try:
if not 0 <= float(fsr_detail) <= 1:
raise ValueError
except ValueError:
fail(f"[RayReconstruction] FsrDetail {fsr_detail!r}: use a number from 0 to 1")
env["D4R_FSR_RR_DETAIL"] = fsr_detail
fsr_denoiser = path_value("RayReconstruction", "FsrDenoiser")
if fsr_denoiser:
env["D4R_FSR_RR_DLL"] = fsr_denoiser
age = get("Latency", "FrameAge")
if age is not None:
@@ -162,6 +190,7 @@ def main(argv):
if value is not None:
env[var] = "1" if value else "0"
value = get("Interop", "MarkerPollUs")
env["D4R_MOTION_THIN_SKY"] = "1" if flag("Interop", "MotionDilationThinSky") else "0"
dilation = get("Interop", "MotionVectorDilation")
if dilation is not None and dilation not in ("0", "1", "2"):
fail("[Interop] MotionVectorDilation must be 0, 1 or 2")
@@ -213,6 +242,11 @@ def main(argv):
if flag("Debug", "Profile"):
env["D4R_PROFILE"] = "1"
every = get("Debug", "ProfileLogEvery")
if every is not None:
if not every.isascii() or not every.isdecimal() or not 1 <= int(every) <= 4294967295:
fail(f"[Debug] ProfileLogEvery must be 1-4294967295, not {every!r}")
env["D4R_PROFILE_LOG_EVERY"] = str(int(every))
if path_value("Debug", "LogDir"):
env["D4R_GAME_LOG_DIR"] = path_value("Debug", "LogDir")
scripts/package_release.sh
+5
View File
@@ -128,6 +128,11 @@ cp -r "$ROCM_RUNTIME/lib" "$STAGE/d4r/rocm/lib"
"$STAGE/d4r/nvcuda.dll" "$STAGE/d4r/zluda/libcuda.so" "$STAGE/d4r/rocm/lib/"*
cp "$ROOT/packaging/d4r.ini" "$STAGE/d4r/d4r.ini"
cp "$ROOT/packaging/d4r-check.sh" "$STAGE/d4r/d4r-check.sh"
# AMD FSR Ray Regeneration 1.2 for [RayReconstruction] Backend = fsr: AMD's unmodified denoiser DLL (MIT)
if [[ -n "${D4R_BUNDLE_FSR_RR_DLL:-}" ]]; then
cp "$D4R_BUNDLE_FSR_RR_DLL" "$STAGE/d4r/amd_fidelityfx_denoiser_dx12.dll"
cp "$ROOT/third_party/fidelityfx/LICENSE.txt" "$STAGE/d4r/licenses/FidelityFX-LICENSE.txt"
fi
if [[ "$VARIANT" != clean ]]; then
: "${D4R_BUNDLE_DLSS:?set D4R_BUNDLE_DLSS}" "${D4R_BUNDLE_TEX:?set D4R_BUNDLE_TEX}"
if [[ "$VARIANT" == full ]]; then
scripts/run_fsr_rr_probe_proton.sh
Executable
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
set -euo pipefail
# Runs build/fsr_rr_probe.exe through a Proton build in its own prefix, with a given vkd3d-proton
# (d3d12.dll + d3d12core.dll) and AMD denoiser DLL copied next to the exe, as a game install would
# have them. Nothing outside APP_DIR and the prefix is touched.
# usage: run_fsr_rr_probe_proton.sh PROTON_DIR VKD3D_DIR DENOISER_DLL OUT_DIR [probe args...]
# Env: D4R_FSR_PROBE_COMPAT prefix (default ${TMPDIR:-/tmp}/d4r-fsr-rr-probe-compat)
# D4R_FSR_PROBE_APP app directory (default OUT_DIR/app)
# DXIL_SPIRV_CONFIG passed through; defaults to wmma_rdna3_workaround (gfx11 WMMA path)
# D4R_FSR_PROBE_SPOOF=1 report an NVIDIA GPU the way d4r's DLSS games run
# (PROTON_FORCE_NVAPI=1 DXVK_NVAPI_GPU_ARCH=AD100)
if [[ $# -lt 4 ]]; then
printf 'usage: %s PROTON_DIR VKD3D_DIR DENOISER_DLL OUT_DIR [probe args...]\n' "$0" >&2
exit 2
fi
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PROTON_DIR="$(realpath "$1")"
VKD3D_DIR="$(realpath "$2")"
DENOISER="$(realpath "$3")"
OUT="$(realpath -m "$4")"
shift 4
winpath() { printf 'Z:%s' "$(printf '%s' "$1" | tr '/' '\\')"; }
[[ -f "$ROOT/build/fsr_rr_probe.exe" ]] || "$ROOT/scripts/build_fsr_rr_probe.sh" >/dev/null
APP="${D4R_FSR_PROBE_APP:-$OUT/app}"
mkdir -p "$APP" "$OUT"
cp -f "$ROOT/build/fsr_rr_probe.exe" "$APP/"
cp -f "$VKD3D_DIR/d3d12.dll" "$VKD3D_DIR/d3d12core.dll" "$APP/"
cp -f "$DENOISER" "$APP/amd_fidelityfx_denoiser_dx12.dll"
{
printf 'proton=%s\n' "$PROTON_DIR"
sha256sum "$APP/d3d12.dll" "$APP/d3d12core.dll" "$APP/amd_fidelityfx_denoiser_dx12.dll" "$APP/fsr_rr_probe.exe"
} > "$OUT/run-manifest.txt"
export STEAM_COMPAT_CLIENT_INSTALL_PATH="${STEAM_COMPAT_CLIENT_INSTALL_PATH:-$HOME/.local/share/Steam}"
export STEAM_COMPAT_DATA_PATH="$(realpath -m "${D4R_FSR_PROBE_COMPAT:-${TMPDIR:-/tmp}/d4r-fsr-rr-probe-compat}")"
mkdir -p "$STEAM_COMPAT_DATA_PATH"
export WINEDLLOVERRIDES="d3d12,d3d12core=n,b${WINEDLLOVERRIDES:+;$WINEDLLOVERRIDES}"
export DXIL_SPIRV_CONFIG="${DXIL_SPIRV_CONFIG-wmma_rdna3_workaround}"
export VKD3D_SHADER_CACHE_PATH="${VKD3D_SHADER_CACHE_PATH:-$(winpath "$OUT")}"
export VKD3D_DEBUG="${VKD3D_DEBUG:-warn}"
if [[ "${D4R_FSR_PROBE_SPOOF:-0}" == 1 ]]; then
export PROTON_FORCE_NVAPI=1 DXVK_NVAPI_GPU_ARCH=AD100
fi
printf 'DXIL_SPIRV_CONFIG=%s spoof=%s\n' "$DXIL_SPIRV_CONFIG" "${D4R_FSR_PROBE_SPOOF:-0}" >> "$OUT/run-manifest.txt"
set +e
"$PROTON_DIR/proton" run "$APP/fsr_rr_probe.exe" "$(winpath "$APP/amd_fidelityfx_denoiser_dx12.dll")" \
"$(winpath "$OUT")" "$@" 2> "$OUT/stderr.log" | tee "$OUT/stdout.log"
status=${PIPESTATUS[0]}
set -e
printf 'exit=%s\n' "$status" >> "$OUT/run-manifest.txt"
exit "$status"
scripts/run_rr_backend_check_proton.sh
Executable
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
set -euo pipefail
# Runs build/rr_backend_check.exe through a Proton build in its own prefix, against a portable d4r
# folder (d4r.ini, d4r_nvngx.dll, nvcuda.dll, ngx/, zluda/, and amd_fidelityfx_denoiser_dx12.dll or
# nvngx_dlssd.dll for the backend under test), with the d4r vkd3d-proton next to the exe.
# The GPU is reported as NVIDIA (PROTON_FORCE_NVAPI=1 DXVK_NVAPI_GPU_ARCH=AD100), as d4r's games run.
# usage: run_rr_backend_check_proton.sh PROTON_DIR VKD3D_DIR D4R_DIR OUT_DIR [check args...]
# Env: D4R_RR_BACKEND (dlss|fsr) and any other shim variable are passed through.
# D4R_RR_CHECK_COMPAT prefix (default ${TMPDIR:-/tmp}/d4r-rr-check-compat)
if [[ $# -lt 4 ]]; then
printf 'usage: %s PROTON_DIR VKD3D_DIR D4R_DIR OUT_DIR [check args...]\n' "$0" >&2
exit 2
fi
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PROTON_DIR="$(realpath "$1")"
VKD3D_DIR="$(realpath "$2")"
D4R_DIR="$(realpath "$3")"
OUT="$(realpath -m "$4")"
shift 4
winpath() { printf 'Z:%s' "$(printf '%s' "$1" | tr '/' '\\')"; }
[[ -f "$ROOT/build/rr_backend_check.exe" ]] || "$ROOT/scripts/build_rr_backend_check.sh" >/dev/null
APP="$OUT/app"
mkdir -p "$APP"
cp -f "$ROOT/build/rr_backend_check.exe" "$APP/"
cp -f "$VKD3D_DIR/d3d12.dll" "$VKD3D_DIR/d3d12core.dll" "$APP/"
{
printf 'proton=%s\nd4r=%s\nbackend=%s\n' "$PROTON_DIR" "$D4R_DIR" "${D4R_RR_BACKEND:-ini}"
sha256sum "$APP/d3d12.dll" "$APP/d3d12core.dll" "$D4R_DIR"/*.dll
} > "$OUT/run-manifest.txt"
export STEAM_COMPAT_CLIENT_INSTALL_PATH="${STEAM_COMPAT_CLIENT_INSTALL_PATH:-$HOME/.local/share/Steam}"
export STEAM_COMPAT_DATA_PATH="$(realpath -m "${D4R_RR_CHECK_COMPAT:-${TMPDIR:-/tmp}/d4r-rr-check-compat}")"
mkdir -p "$STEAM_COMPAT_DATA_PATH"
export WINEDLLOVERRIDES="d3d12,d3d12core=n,b${WINEDLLOVERRIDES:+;$WINEDLLOVERRIDES}"
export DXIL_SPIRV_CONFIG="${DXIL_SPIRV_CONFIG-wmma_rdna3_workaround}"
export VKD3D_SHADER_CACHE_PATH="${VKD3D_SHADER_CACHE_PATH:-$(winpath "$OUT")}"
export PROTON_FORCE_NVAPI=1 DXVK_NVAPI_GPU_ARCH=AD100
export D4R_SHIM_LOG="$(winpath "$OUT/d4r_nvngx.log")"
set +e
"$PROTON_DIR/proton" run "$APP/rr_backend_check.exe" "$(winpath "$D4R_DIR/d4r_nvngx.dll")" "$(winpath "$OUT")" "$@" \
> "$OUT/stdout.log" 2> "$OUT/stderr.log"
status=$?
set -e
printf 'exit=%s\n' "$status" >> "$OUT/run-manifest.txt"
grep -E '^(check=|overall=|tonemapped)' "$OUT/stdout.log" || true
exit "$status"
scripts/summarize_d4r_profile.py
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""Summarize warm D4R_PROFILE samples without merging feature instances or sizes.
GPU pass fields and delayed vk_* fields are summarized independently. Their
per-line difference is deliberately not calculated; they describe different frames.
"""
import argparse
from collections import defaultdict
import json
import math
from pathlib import Path
import re
import statistics
GROUP_FIELDS = ('render', 'output', 'preset', 'quality', 'flags', 'input_wait',
'feature', 'native_field', 'native_substitute', 'wire_requested', 'gpu_completion', 'vk_output_mode', 'vk_input_mode')
FIELDS = ('vk_total', 'dlss_gpu', 'vk_input', 'vk_dlss_wait', 'vk_output',
'native_motion_gpu', 'wire_repair_gpu', 'marker_wait', 'pipeline',
'throughput_interval')
def summarize(text, min_frame=300, since_clock=None, until_clock=None):
groups = defaultdict(list)
instance = 0
for line in text.splitlines():
if 'NVSDK_NGX_D3D12_CreateFeature(feature=' in line:
instance += 1
if 'D4R_PROFILE frame=' not in line:
continue
if since_clock is not None or until_clock is not None:
clock = re.search(r'^\[\s*([0-9.]+)\]', line)
if clock is None:
continue
seconds = float(clock[1])
if since_clock is not None and seconds < since_clock:
continue
if until_clock is not None and seconds > until_clock:
continue
values = dict(re.findall(r'(\w+)=([^\s]+)', line))
if int(values['frame']) < min_frame or values.get('reset') != '0':
continue
key = (instance, *(values.get(k, 'unknown') for k in
GROUP_FIELDS))
groups[key].append(values)
result = []
for key, samples in groups.items():
row = dict(zip(('feature_instance', *GROUP_FIELDS), key))
row.update(samples=len(samples), first_frame=int(samples[0]['frame']), last_frame=int(samples[-1]['frame']))
row['timings_ms'] = {}
for field in FIELDS:
values = [float(s.get(field, '-1')) for s in samples]
valid = [v for v in values if math.isfinite(v) and v >= 0]
row['timings_ms'][field] = (dict(n=len(valid), median=statistics.median(valid),
min=min(valid), max=max(valid)) if valid else None)
result.append(row)
return result
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('log', type=Path)
parser.add_argument('--min-frame', type=int, default=300)
parser.add_argument('--since-clock', type=float, help='minimum bracketed log-clock seconds')
parser.add_argument('--until-clock', type=float, help='maximum bracketed log-clock seconds')
args = parser.parse_args()
print(json.dumps(summarize(args.log.read_text(errors='replace'), args.min_frame,
args.since_clock, args.until_clock), indent=2))
if __name__ == '__main__':
main()
tests/test_accuracy.py
+11
View File
@@ -174,6 +174,17 @@ int main(int argc, char** argv) {
self.config("[Interop]\nMotionVectorDilation = 2\n",
{"D4R_MOTION_DILATION": "0"}).stdout)
def test_thin_sky_opt_in_and_environment_precedence(self):
for value, expected in (("false","0"),("true","1"),("auto","0")):
self.assertIn(f"export D4R_MOTION_THIN_SKY={expected}",
self.config(f"[Interop]\nMotionDilationThinSky = {value}\n").stdout)
self.config("[Interop]\nMotionDilationThinSky = maybe\n", expected=2)
self.assertNotIn("export D4R_MOTION_THIN_SKY",
self.config("[Interop]\nMotionDilationThinSky = true\n",
{"D4R_MOTION_THIN_SKY":"0"}).stdout)
for file in ("config/d4r.ini.default", "packaging/d4r.ini"):
self.assertRegex((ROOT/file).read_text(), r"(?m)^MotionDilationThinSky\s*=\s*false$")
def test_shipped_configs_default_off(self):
for file in ("config/d4r.ini.default", "packaging/d4r.ini"):
self.assertRegex((ROOT / file).read_text(), r"(?m)^PreferAccuracy\s*=\s*false$")
tests/test_asi_setup.py
+3 -1
View File
@@ -185,11 +185,13 @@ int wmain(int argc, wchar_t** argv) {
(runtime / 'ngx').mkdir(parents=True)
for name in ('nvngx.dll', 'd4r.ini'):
(runtime / name).touch()
dlss, ngx = b'verified DLSS fixture', b'verified NGX fixture'
dlss, dlssd, ngx = b'verified DLSS fixture', b'verified DLSS RR fixture', b'verified NGX fixture'
(runtime / 'nvngx_dlss.dll').write_bytes(dlss)
(runtime / 'nvngx_dlssd.dll').write_bytes(dlssd)
(runtime / 'ngx/_nvngx.dll').write_bytes(ngx)
script = (ROOT / 'install.sh').read_text()
replacements = {'DLSS_SHA256': hashlib.sha256(dlss).hexdigest(),
'DLSSD_SHA256': hashlib.sha256(dlssd).hexdigest(),
'NGX_SHA256': hashlib.sha256(ngx).hexdigest()}
script = '\n'.join(key + '=' + replacements[key] if key in replacements else line
for line in script.splitlines() for key in [line.split('=', 1)[0]]) + '\n'
tests/test_fused_vram_output.py
+110
View File
@@ -0,0 +1,110 @@
"""Exercise the actual optional output recorder's lifetime and fallback paths."""
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
class FusedVramOutputTests(unittest.TestCase):
def test_output_lifetime_and_fallback(self):
shim=(ROOT/'tools/d4r_nvngx_shim.cpp').read_text()
start=shim.index('static bool record_feature_vram_output(')
end=shim.index('// --- Split frames',start)
input_start=shim.index('static bool ensure_fused_color(')
input_end=shim.index('// Records the input copies',input_start)
program=r'''
#include <cassert>
#include <cstddef>
#include <cstdint>
#include <vector>
using VkImage=uint64_t;using VkImageView=uint64_t;using VkBuffer=uint64_t;
using VkCommandBuffer=uint64_t;using D3D12_RESOURCE_STATES=int;
constexpr int VK_SUCCESS=0,VK_NULL_HANDLE=0,VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO=1,
VK_STRUCTURE_TYPE_MEMORY_BARRIER=2,VK_IMAGE_VIEW_TYPE_2D=3,VK_FORMAT_B10G11R11_UFLOAT_PACK32=4,
VK_IMAGE_ASPECT_COLOR_BIT=1,VK_ACCESS_MEMORY_WRITE_BIT=1,VK_ACCESS_SHADER_READ_BIT=2,
VK_ACCESS_SHADER_WRITE_BIT=4,VK_ACCESS_MEMORY_READ_BIT=8,VK_PIPELINE_STAGE_ALL_COMMANDS_BIT=1,
VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT=2,DXGI_FORMAT_R11G11B10_FLOAT=26,
D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS=4,D3D12_RESOURCE_FLAG_DENY_SHADER_RESOURCE=8,
DXGI_FORMAT_R8G8B8A8_UNORM=28,VK_FORMAT_R8G8B8A8_UNORM=5,D3D12_RESOURCE_STATE_UNORDERED_ACCESS=8,
D3D12_RESOURCE_STATE_COPY_DEST=16;
#define SUCCEEDED(value) ((value)>=0)
struct Range {int aspect,baseMip,levels,baseLayer,layers;};
struct VkImageViewCreateInfo {int sType;VkImage image=0;int viewType=0,format=0;Range subresourceRange={};};
struct VkMemoryBarrier {int sType;void* next;int src,dst;};
struct Desc {int Format=26,Flags=4;};
struct ID3D12Resource {Desc desc;int refs=0;Desc GetDesc(){return desc;}void AddRef(){++refs;}};
struct ID3D12GraphicsCommandList {};
struct Set {int pool=0;};
struct FusedColorInput {VkImage image=0;VkImageView view=0;VkBuffer buffer=0;Set set;ID3D12Resource* owner=nullptr;uint32_t inputMode=5;};
struct VramBuffer {VkBuffer buffer=1;size_t bytes=800;};
struct VramCopy {bool convert=true;unsigned width=10,height=10;VkImage image=10;};
struct Feature {bool fusedOutputAllowed=true,recordOutputFused=false,fusedColorAllowed=true;
struct {VramBuffer vram[1];struct {size_t size(){return 800;}} host[1];} inputs[3];
FusedColorInput fusedColor[3];std::vector<FusedColorInput> retiredFusedColor;struct {VramBuffer vram;} outputs[4];FusedColorInput fusedOutput[4];std::vector<FusedColorInput> retiredFusedOutput;int outputConversion=0;};
static unsigned enabled=1;
static bool g_fusedColorReady=true,viewOk=true,setOk=true,beginOk=true,endOk=true;
static int views=0,destroyed=0,frees=0,records=0,legacy=0;
static unsigned env_uint(const char*,unsigned){return enabled;}
static void logf(const char*,...) {}
static std::vector<int> transitions;
static void transition(ID3D12GraphicsCommandList*,ID3D12Resource*,int,int after){transitions.push_back(after);}
static bool record_vram_output(ID3D12GraphicsCommandList*,const VramBuffer&,const VramCopy&,int){++legacy;return true;}
struct Interop {int BeginVkCommandBufferInterop(ID3D12GraphicsCommandList*,VkCommandBuffer* c){*c=1;return beginOk?0:-1;}int EndVkCommandBufferInterop(ID3D12GraphicsCommandList*){return endOk?0:-1;}} interop;
static struct {
int device=0;Interop* interop=&::interop;
int createView(int,const VkImageViewCreateInfo*,void*,VkImageView* out){++views;if(viewOk)*out=5;return viewOk?0:-1;}
void destroyView(int,VkImageView,void*){++destroyed;}
void barrier(VkCommandBuffer,int,int,int,int,const VkMemoryBarrier*,int,void*,int,void*){}
} g_vk;
static struct {
bool set(Set& s,VkImageView,int,VkBuffer,size_t,bool){s.pool=1;return setOk;}
void free(Set& s){++frees;s={};}
void record(VkCommandBuffer,const Set&,unsigned,unsigned){++records;}
} g_fusedColorStage;
''' + shim[input_start:input_end] + shim[start:end] + r'''
int main() {
ID3D12GraphicsCommandList list;ID3D12Resource output,changed;VramCopy copy;Feature f;
enabled=false;assert(record_feature_vram_output(f,0,&list,&output,8,copy));assert(legacy==1 && views==0 && records==0);
enabled=true;assert(record_feature_vram_output(f,0,&list,&output,8,copy));assert(output.refs==1 && views==1 && records==1 && legacy==1);
assert(record_feature_vram_output(f,0,&list,&output,8,copy));assert(output.refs==1 && views==1 && records==2);
assert(record_feature_vram_output(f,0,&list,&changed,8,copy));assert(changed.refs==1 && output.refs==1 && f.retiredFusedOutput.size()==1);
Feature missingUav;ID3D12Resource noUav;noUav.desc.Flags=0;int old=legacy;
assert(record_feature_vram_output(missingUav,0,&list,&noUav,8,copy));assert(legacy==old+1 && !noUav.refs);
Feature small;small.outputs[0].vram.bytes=799;old=legacy;
assert(record_feature_vram_output(small,0,&list,&output,8,copy));assert(legacy==old+1);
Feature packed;copy.convert=false;old=legacy;
assert(record_feature_vram_output(packed,0,&list,&output,8,copy));assert(legacy==old+1);copy.convert=true;
Feature noView;viewOk=false;old=legacy;
assert(record_feature_vram_output(noView,0,&list,&output,8,copy));assert(legacy==old+1 && !noView.fusedOutputAllowed);viewOk=true;
Feature noSet;setOk=false;int released=destroyed;old=legacy;
assert(record_feature_vram_output(noSet,0,&list,&output,8,copy));assert(legacy==old+1 && destroyed==released+1 && !noSet.fusedOutputAllowed);setOk=true;
Feature noBegin;beginOk=false;old=legacy;int oldRecords=records;
assert(record_feature_vram_output(noBegin,0,&list,&output,8,copy));assert(legacy==old+1 && records==oldRecords && !noBegin.fusedOutputAllowed);beginOk=true;
Feature noEnd;endOk=false;old=legacy;
assert(record_feature_vram_output(noEnd,0,&list,&output,8,copy));assert(legacy==old+1 && !noEnd.fusedOutputAllowed);endOk=true;
Feature bounded;bounded.retiredFusedOutput.resize(64);old=legacy;int oldViews=views;
assert(record_feature_vram_output(bounded,0,&list,&output,8,copy));assert(legacy==old+1 && views==oldViews && !bounded.fusedOutputAllowed);
enabled=2;Feature alternating;old=legacy;
assert(record_feature_vram_output(alternating,0,&list,&output,8,copy,0));assert(legacy==old+1 && !alternating.recordOutputFused);
assert(record_feature_vram_output(alternating,0,&list,&output,8,copy,300));assert(legacy==old+1 && alternating.recordOutputFused);
assert(record_feature_vram_output(alternating,0,&list,&output,8,copy,600));assert(legacy==old+2 && !alternating.recordOutputFused);
enabled=1;Feature input;ID3D12Resource r11;
assert(ensure_fused_color(input,0,&r11,copy,0));assert(input.fusedColor[0].inputMode==1 && r11.refs==1);
int inputViews=views;assert(ensure_fused_color(input,0,&r11,copy,0));assert(views==inputViews && r11.refs==1);
enabled=2;assert(!ensure_fused_color(input,0,&r11,copy,300));assert(ensure_fused_color(input,0,&r11,copy,600));
assert(!ensure_fused_color(input,0,&r11,copy,1200));assert(views==inputViews);
enabled=1;ID3D12Resource rgba8;rgba8.desc.Format=28;Feature rgbaInput;
assert(ensure_fused_color(rgbaInput,0,&rgba8,copy,0));assert(rgbaInput.fusedColor[0].inputMode==5);
ID3D12Resource denied;denied.desc.Flags=8;assert(!ensure_fused_color(input,0,&denied,copy));
Feature limitedInput;limitedInput.retiredFusedColor.resize(64);
assert(!ensure_fused_color(limitedInput,0,&r11,copy) && !limitedInput.fusedColorAllowed);
assert(transitions.back()==8); // every route restores caller's output state
}
'''
with tempfile.TemporaryDirectory() as temp:
runner=Path(temp)/'fused-output'
subprocess.run(['c++','-std=c++17','-Wall','-Wextra','-Werror','-Wno-missing-field-initializers',
'-x','c++','-','-o',str(runner)],input=program,text=True,check=True,capture_output=True)
subprocess.run([str(runner)],check=True,timeout=5)
tests/test_motion_geometry.py
+13
View File
@@ -12,6 +12,7 @@ class MotionGeometryTests(unittest.TestCase):
source = r'''
#include "d4r_motion_dilation.h"
#include <cassert>
#include <initializer_list>
int main() {
D4rMotionDilationParams p = {};
p.render_width = 854; p.render_height = 480;
@@ -21,6 +22,18 @@ int main() {
assert(d4r_motion_dilation_geometry(&low, 0xb, 2560, 1440));
assert(low.motion_width == 854 && low.motion_height == 480);
assert(low.motion_pitch == 4096 && low.jitter_x == 0 && low.jitter_y == 0);
for (unsigned radius : {1u, 2u}) {
low.radius = radius;
assert(d4r_motion_dilation_lds_eligible(&low));
auto fallback = low; fallback.jitter_x = 0.25f;
assert(!d4r_motion_dilation_lds_eligible(&fallback));
fallback = low; ++fallback.motion_height;
assert(!d4r_motion_dilation_lds_eligible(&fallback));
fallback = low; ++fallback.motion_width;
assert(!d4r_motion_dilation_lds_eligible(&fallback));
}
low.radius = 0; assert(!d4r_motion_dilation_lds_eligible(&low));
low.radius = 3; assert(!d4r_motion_dilation_lds_eligible(&low));
low = p;
assert(d4r_motion_dilation_geometry(&low, 0x4b, 2560, 1440));
low = p; low.motion_width = 853;
tests/test_motion_kernel_gpu.py
+149
View File
@@ -121,3 +121,152 @@ class MotionKernelGpuTests(unittest.TestCase):
_, sx, sy = candidates[0]
expected[y*stride+x] = motion[sy*stride+sx]
self.assertEqual((output/'alloc-2.bin').read_bytes(), struct.pack(f'<{len(expected)}I', *expected))
@unittest.skipUnless(os.environ.get('D4R_RUN_GPU_TESTS') == '1', 'opt-in GPU test')
class MotionLdsKernelGpuTests(unittest.TestCase):
def run_kernel(self, name, width, height, stride, allocation_height, depth, motion, radius, reverse):
runner = os.environ['D4R_DUMP_RUNNER']
kernel = ROOT / 'build/wine-nvcuda/generated' / (os.environ.get('D4R_GPU_ARCH', 'gfx1101') + '.hsaco')
# Guard the tail beyond the allocation too. Generic output is pre-copied,
# matching the bridge's original full-plane copy followed by dilation.
guard = [0xA5A5A5A5] * 32
original = motion + guard
arrays = [struct.pack(f'<{len(depth)}f', *depth),
struct.pack(f'<{len(original)}I', *original),
struct.pack(f'<{len(original)}I', *(original if name == 'd4r_motion_dilate' else [0xA5A5A5A5]*len(original)))]
addresses = (0x100000000, 0x200000000, 0x300000000)
args = struct.pack('<5Q4I2f2I', *addresses, stride*4, stride*4,
width, height, width, height, 0., 0., radius, reverse)
if name.startswith('d4r_motion_dilate_lds'):
args += struct.pack('<I', allocation_height)
grid = f'{(stride+15)//16} {(allocation_height+15)//16} 1 256 1 1 0'
else:
grid = f'{(width*height+127)//128} 1 1 128 1 1 0'
with tempfile.TemporaryDirectory() as t:
directory = Path(t); output = directory/'out'; output.mkdir()
(directory/'args.bin').write_bytes(args)
manifest = [f'kernel {name}', f'launch {grid}', f'args {len(args)}']
for i, data in enumerate(arrays):
(directory/f'alloc-{i}.bin').write_bytes(data)
manifest += [f'alloc {i} {hex(addresses[i])} {len(data)}', f'pointer {i*8} {i} 0']
(directory/'manifest.txt').write_text('\n'.join(manifest)+'\n')
subprocess.run([runner, str(kernel), name, str(directory), str(output), '1'],
check=True, capture_output=True, timeout=30)
for i in (0, 1):
self.assertEqual((output/f'alloc-{i}.bin').read_bytes(), arrays[i], 'input mutated')
result = (output/'alloc-2.bin').read_bytes()
self.assertEqual(result[-128:], struct.pack('<32I', *guard), 'tail guard overwritten')
values = struct.unpack(f'<{len(original)}I', result)
for y in range(allocation_height):
for x in range(stride):
if x >= width or y >= height:
self.assertEqual(values[y*stride+x], motion[y*stride+x], 'padding changed')
return result
def test_image_edges_block_edges_and_lexicographic_sky_ties(self):
width, height, stride, allocation_height = 35, 33, 48, 36
# Equal-distance sources straddle x=16/y=16 tile boundaries. Include
# corners, circle boundary, and same-x/different-y tie candidates.
source_sets = [
[(0, 0), (34, 32), (13, 16), (19, 16), (8, 31)],
[(0, 0), (34, 32), (16, 13), (16, 19), (8, 31)],
]
for sources in source_sets:
for reverse in (False, True):
for radius in (1, 2):
with self.subTest(reverse=reverse, radius=radius, sources=sources):
depth = [float('nan')] * (stride*allocation_height)
motion = list(range(stride*allocation_height))
for y in range(height):
for x in range(width):
depth[y*stride+x] = 0. if reverse else 1.
for x, y in sources:
depth[y*stride+x] = .9 if reverse else .1
output = self.run_kernel('d4r_motion_dilate_lds', width, height, stride, allocation_height,
depth, motion, radius, reverse)
values = struct.unpack(f'<{len(output)//4}I', output)
for y in range(height):
for x in range(width):
nearby = sorted(((x-sx)**2+(y-sy)**2, sx, sy) for sx, sy in sources)
expected = motion[y*stride+x]
if nearby[0][0] <= 64:
_, sx, sy = nearby[0]
expected = motion[sy*stride+sx]
self.assertEqual(values[y*stride+x], expected, (x, y))
def test_randomized_bit_exact_generic_equivalence(self):
import random
rng = random.Random(0xD4A)
for width, height in ((1, 1), (2, 3), (15, 17), (17, 31), (33, 19), (65, 37)):
stride, allocation_height = ((width+7)//8+1)*8, height+3
for scene in ('empty', 'sparse', 'dense', 'special'):
forward = [float('nan')] * (stride*allocation_height)
for y in range(height):
for x in range(width):
d = 1.
if scene == 'dense' or (scene == 'sparse' and rng.random() < .12):
d = rng.choice((.1, .5, .9, .99))
elif scene == 'special':
d = rng.choice((0., 1., 1.e-7, 1.-1.e-7, float('nan'), float('inf'), -float('inf'), .98, .99999994))
forward[y*stride+x] = d
motion = [rng.getrandbits(32) for _ in forward]
for reverse in (False, True):
depth = [1.-d for d in forward] if reverse else forward
for radius in (1, 2):
with self.subTest(size=(width, height), scene=scene, reverse=reverse, radius=radius):
old = self.run_kernel('d4r_motion_dilate', width, height, stride, allocation_height,
depth, motion, radius, reverse)
new = self.run_kernel('d4r_motion_dilate_lds', width, height, stride, allocation_height,
depth, motion, radius, reverse)
self.assertEqual(old, new, 'LDS must have zero mismatched bytes')
def test_thin_sky_gate_matches_global_run_reference(self):
# Wide roof plus a long pole, horizontal wire, diagonal and a short tip.
# Moving the shapes across x/y=16 exercises the extra morphology halo.
for shift in (0, 3):
width, height, stride, ah = 47, 39, 56, 42
foreground = {(x, y) for y in range(24, 39) for x in range(20+shift, 47)}
foreground |= {(8+shift, y) for y in range(2, 29)}
foreground |= {(x, 8+shift) for x in range(2, 38)}
foreground |= {(x, x//2+12) for x in range(2, 20)}
def short_run(x, y, dx, dy):
length = 1
for sign in (-1, 1):
step = 1
while (x+sign*step*dx, y+sign*step*dy) in foreground:
length += 1
step += 1
return length <= 3
eligible = [(x, y) for x, y in foreground
if short_run(x,y,1,0) or short_run(x,y,0,1)]
self.assertIn((8+shift, 18), eligible, 'long pole must survive')
self.assertIn((18, 8+shift), eligible, 'horizontal wire must survive')
self.assertNotIn((30+shift, 28), eligible, 'broad roof must be rejected')
for reverse in (False, True):
for radius in (1, 2):
with self.subTest(shift=shift, reverse=reverse, radius=radius):
depth = [float('nan')]*(stride*ah)
motion = list(range(stride*ah))
for y in range(height):
for x in range(width):
depth[y*stride+x] = (.9 if reverse else .1) if (x,y) in foreground else (0. if reverse else 1.)
out = self.run_kernel('d4r_motion_dilate_lds_thin', width, height, stride, ah,
depth, motion, radius, reverse)
values = struct.unpack(f'<{len(out)//4}I',out)
for y in range(height):
for x in range(width):
bx, by = x, y
# Opaque dilation is deliberately retained even where sky fill is gated.
if (x,y) not in foreground:
found = False
for dy in range(-radius,radius+1):
for dx in range(-radius,radius+1):
nx, ny = max(0,min(width-1,x+dx)), max(0,min(height-1,y+dy))
if not found and (nx,ny) in foreground:
bx, by = nx, ny
found = True
near = sorted(((x-sx)**2+(y-sy)**2,sx,sy) for sx,sy in eligible)
if near and near[0][0] <= 64:
_, bx, by = near[0]
self.assertEqual(values[y*stride+x], motion[by*stride+bx], (x,y))
tests/test_motion_submission.py
+75
View File
@@ -0,0 +1,75 @@
"""Dilation must not enter GPU input waits while lazy resources can synchronize."""
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
class MotionSubmissionTests(unittest.TestCase):
def test_gpu_wait_requires_a_warm_compatible_lds_slot(self):
source = (ROOT / 'tools/d4r_nvngx_shim.cpp').read_text()
begin = source.index('static bool motion_dilation_parameters(')
end = source.index('static CudaObject prepare_dilated_motion(', begin)
program = r'''
#include "d4r_motion_dilation.h"
#include <cassert>
#include <cmath>
#include <cstddef>
using CudaObject = uint64_t;
static unsigned radius = 1;
static unsigned env_uint(const char*, unsigned) { return radius; }
struct HostPlane { unsigned width=20, height=20; size_t rowBytes=128; size_t size() const { return rowBytes*height; } };
struct InputSlot {
HostPlane host[3];
struct { uint64_t device=1; } vram[3];
uint64_t dilatedMotion=3, dilatedTexture=4;
size_t dilatedBytes=2560, dilatedPitch=128;
unsigned dilatedWidth=20, dilatedHeight=20;
bool dilationEnqueued=false;
};
struct Feature { unsigned flags=2, outWidth=40, outHeight=40; bool linearInputs=true; };
struct FrameParams {
unsigned renderWidth=17, renderHeight=19;
float jitterX=.25f, jitterY=-.25f;
bool vram=true;
unsigned colorBaseX=0, colorBaseY=0, depthBaseX=0, depthBaseY=0,
mvBaseX=0, mvBaseY=0, outputBaseX=0, outputBaseY=0;
};
struct { struct { void* dilateMotion=(void*)1; void* dilateMotionPadded=(void*)1; } cu; } g;
''' + source[begin:end] + r'''
int main() {
Feature f; InputSlot slot; FrameParams p;
assert(!motion_dilation_gpu_wait_ready(f,slot,p));
slot.dilationEnqueued=true;
assert(motion_dilation_gpu_wait_ready(f,slot,p));
radius=2; assert(motion_dilation_gpu_wait_ready(f,slot,p));
auto changed=slot; changed.host[2].rowBytes+=4;
assert(!motion_dilation_gpu_wait_ready(f,changed,p));
changed=slot; ++changed.host[2].width;
assert(!motion_dilation_gpu_wait_ready(f,changed,p));
changed=slot; ++changed.host[2].height;
assert(!motion_dilation_gpu_wait_ready(f,changed,p));
changed=slot; changed.dilatedTexture=0;
assert(!motion_dilation_gpu_wait_ready(f,changed,p));
changed=slot; changed.dilatedBytes=0;
assert(!motion_dilation_gpu_wait_ready(f,changed,p));
g.cu.dilateMotionPadded=nullptr;
assert(!motion_dilation_gpu_wait_ready(f,slot,p));
g.cu.dilateMotionPadded=(void*)1;
p.depthBaseX=1; assert(!motion_dilation_gpu_wait_ready(f,slot,p));
p.depthBaseX=0; f.flags=6; assert(!motion_dilation_gpu_wait_ready(f,slot,p));
f.flags=0; f.outWidth=20; f.outHeight=20;
assert(!motion_dilation_gpu_wait_ready(f,slot,p)); // mapped display-res fallback
f.flags=2; p.renderWidth=21;
assert(!motion_dilation_gpu_wait_ready(f,slot,p)); // invalid allocation
radius=0; assert(motion_dilation_gpu_wait_ready(f,slot,p)); // dilation off
}
'''
with tempfile.TemporaryDirectory() as temporary:
runner = Path(temporary)/'submission'
subprocess.run(['c++', '-std=c++17', '-Wall', '-Wextra', '-Werror', '-I', str(ROOT/'tools'),
'-x', 'c++', '-', '-o', str(runner)], input=program, text=True,
check=True, capture_output=True)
subprocess.run([str(runner)], check=True, timeout=5)
tests/test_native_aligned_gpu.py
+87
View File
@@ -0,0 +1,87 @@
"""Jitter-aligned native 3x3 motion without the extended halo."""
import math
import os
from pathlib import Path
import struct
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
@unittest.skipUnless(os.environ.get('D4R_RUN_GPU_TESTS') == '1', 'opt-in GPU test')
class NativeAlignedGpuTests(unittest.TestCase):
def test_jitter_nearest_ties_partial_tiles_and_padding(self):
runner = os.environ['D4R_DUMP_RUNNER']
kernel = Path(os.environ.get('D4R_NATIVE_TEST_KERNEL', str(ROOT/'build/wine-nvcuda/generated'/
(os.environ.get('D4R_GPU_ARCH', 'gfx1101')+'.hsaco'))))
w, h, rw, rh = 33, 35, 11, 12
dp, gp, op = w+3, w+5, w+7
sentinel = 0xa5a5a5a5
f32 = lambda x: struct.unpack('<f', struct.pack('<f', x))[0]
depth = [f32(.1+((x*3+y*7)%5)*.1) for y in range(h) for x in range(w)]
geometry = [struct.unpack('<I', struct.pack('<2e', .001*x, .001*y))[0] for y in range(h) for x in range(w)]
for jx, jy in ((0., 0.), (.25, -.25), (-.5, .5), (1., -1.)):
with self.subTest(jitter=(jx, jy)), tempfile.TemporaryDirectory() as tmp:
d = Path(tmp); out = d/'out'; out.mkdir()
padded_d = [float('nan')]*(dp*h)
padded_g = [sentinel]*(gp*h)
expected = [sentinel]*(op*h)
for y in range(h):
for x in range(w):
padded_d[y*dp+x] = depth[y*w+x]
padded_g[y*gp+x] = geometry[y*w+x]
cx = min(w-1, max(0, math.floor(f32(f32(x+.5)+f32(f32(jx*w)/rw)))))
cy = min(h-1, max(0, math.floor(f32(f32(y+.5)+f32(f32(jy*h)/rh)))))
best, bx, by = depth[cy*w+cx], cx, cy
for dy in (-1, 0, 1):
for dx in (-1, 0, 1):
nx, ny = cx+dx, cy+dy
if 0 <= nx < w and 0 <= ny < h and depth[ny*w+nx] < best:
best, bx, by = depth[ny*w+nx], nx, ny
expected[y*op+x] = geometry[by*w+bx]
arrays = [struct.pack('<'+'f'*len(padded_d), *padded_d),
struct.pack('<'+'I'*len(padded_g), *padded_g), bytes(1136), bytes(64),
struct.pack('<'+'I'*(op*h), *([sentinel]*(op*h)))]
addresses = [(i+1)*0x100000000 for i in range(5)]
args = struct.pack('<8Q4I2f', *addresses, dp*4, gp*4, op*4, w, h, rw, rh, jx, jy)
(d/'args.bin').write_bytes(args)
manifest = ['kernel d4r_native_motion_compose_aligned', 'launch 3 3 1 256 1 1 0', 'args 88']
for i, data in enumerate(arrays):
(d/f'alloc-{i}.bin').write_bytes(data)
manifest += [f'alloc {i} {hex(addresses[i])} {len(data)}', f'pointer {i*8} {i} 0']
(d/'manifest.txt').write_text('\n'.join(manifest)+'\n')
subprocess.run([runner, str(kernel), 'd4r_native_motion_compose_aligned', str(d), str(out), '1'],
check=True, capture_output=True)
for i in range(4):
self.assertEqual((out/f'alloc-{i}.bin').read_bytes(), arrays[i])
self.assertEqual((out/'alloc-4.bin').read_bytes(), struct.pack('<'+'I'*len(expected), *expected))
def test_hybrid_preserves_game_vectors_away_from_silhouettes(self):
runner = os.environ['D4R_DUMP_RUNNER']
kernel = ROOT/'build/wine-nvcuda/generated'/(os.environ.get('D4R_GPU_ARCH', 'gfx1101')+'.hsaco')
w,h,rw,rh=33,35,11,12
pitch=(w+3)*4; gp=(rw+3)*4; sentinel=0xa5a5a5a5
game=[struct.unpack('<I',struct.pack('<2e',.001*x,.001*y))[0] for y in range(rh) for x in range(rw)]
padded_game=[sentinel]*((rw+3)*rh)
for y in range(rh): padded_game[y*(rw+3):y*(rw+3)+rw]=game[y*rw:(y+1)*rw]
for depth in (1.,.5):
with self.subTest(depth=depth),tempfile.TemporaryDirectory() as tmp:
d=Path(tmp);out=d/'out';out.mkdir()
arrays=[struct.pack('<f',depth)*(w+3)*h,struct.pack('<I',0x3e003e00)*(w+3)*h,bytes(1136),bytes(64),
struct.pack('<I',sentinel)*(w+3)*h,struct.pack('<'+'I'*len(padded_game),*padded_game)]
addresses=[(i+1)*0x100000000 for i in range(6)]
args=struct.pack('<8Q4I2f2Q',*addresses[:5],pitch,pitch,pitch,w,h,rw,rh,.25,-.25,addresses[5],gp)
(d/'args.bin').write_bytes(args)
manifest=['kernel d4r_native_motion_compose_hybrid','launch 3 3 1 256 1 1 0','args 104']
for i,data in enumerate(arrays):
(d/f'alloc-{i}.bin').write_bytes(data)
manifest += [f'alloc {i} {hex(addresses[i])} {len(data)}',f'pointer {88 if i==5 else i*8} {i} 0']
(d/'manifest.txt').write_text('\n'.join(manifest)+'\n')
subprocess.run([runner,str(kernel),'d4r_native_motion_compose_hybrid',str(d),str(out),'1'],check=True,capture_output=True)
for i in (0,1,2,3,5): self.assertEqual((out/f'alloc-{i}.bin').read_bytes(),arrays[i])
expected=[sentinel]*((w+3)*h)
for y in range(h):
for x in range(w): expected[y*(w+3)+x]=game[int((y+.5)*rh/h)*rw+int((x+.5)*rw/w)]
self.assertEqual((out/'alloc-4.bin').read_bytes(),struct.pack('<'+'I'*len(expected),*expected))
tests/test_native_motion.py
+65
View File
@@ -0,0 +1,65 @@
"""Native compositor ABI and rejection of unsafe buffer contracts."""
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
class NativeMotionTests(unittest.TestCase):
def test_buffer_ranges_and_pitches(self):
source = r'''
#include "d4r_native_motion.h"
#include <assert.h>
#include <string.h>
int main(void) {
D4rNativeMotionParams p; memset(&p,0,sizeof(p));
p.width=2560; p.height=1440; p.render_width=853; p.render_height=480;
p.depth=0x10000000; p.geometry=0x20000000; p.scene=0x30000000;
p.matrix=0x40000000; p.output=0x50000000;
p.depth_pitch=10240; p.geometry_pitch=10244; p.output_pitch=10248;
assert(sizeof(p)==80 && d4r_native_motion_parameters_valid(&p));
assert(!d4r_native_motion_parameters_valid(0));
D4rNativeMotionParams q=p;
q.output=q.geometry+4; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.output=q.scene+16; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.output=q.matrix-16; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.depth=UINT64_MAX-16; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.output_pitch=UINT64_MAX-3; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.geometry_pitch=10239; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.depth_pitch=10241; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.render_width=2561; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.height=0; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.scene=0; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.matrix+=2; assert(!d4r_native_motion_parameters_valid(&q));
q=p; q.height=1; q.render_height=1; q.output_pitch=10240; assert(d4r_native_motion_parameters_valid(&q));
/* Adjacent, disjoint ranges are valid; output padding is not written. */
q=p; q.output=q.geometry+(q.height-1)*q.geometry_pitch+q.width*4;
assert(d4r_native_motion_parameters_valid(&q));
D4rNativeMotionHybridParams hybrid={{p,0,0},0x60000000,853*4};
assert(sizeof(hybrid)==104 && d4r_native_motion_hybrid_parameters_valid(&hybrid));
D4rNativeMotionHybridParams bad=hybrid;bad.game_motion=bad.native.base.output;
assert(!d4r_native_motion_hybrid_parameters_valid(&bad));
bad=hybrid;bad.game_motion_pitch=853*4-1;assert(!d4r_native_motion_hybrid_parameters_valid(&bad));
bad=hybrid;bad.native.jitter_x=2;assert(!d4r_native_motion_hybrid_parameters_valid(&bad));
bad=hybrid;bad.game_motion=UINT64_MAX-16;assert(!d4r_native_motion_hybrid_parameters_valid(&bad));
bad=hybrid;bad.game_motion_pitch=UINT64_MAX-3;assert(!d4r_native_motion_hybrid_parameters_valid(&bad));
D4rNativeMotionPreparedParams prep={{p,0,0},0x70000000,0x80000000,1,0};
assert(sizeof(prep)==112 && d4r_native_motion_prepared_parameters_valid(&prep));
D4rNativeMotionPreparedParams no=prep;no.thin_mask=0;assert(!d4r_native_motion_prepared_parameters_valid(&no));
no=prep;no.wire_axes=no.thin_mask;assert(!d4r_native_motion_prepared_parameters_valid(&no));
no=prep;no.wire_axes=no.native.base.output;assert(!d4r_native_motion_prepared_parameters_valid(&no));
no=prep;no.thin_mask=UINT64_MAX-3;assert(!d4r_native_motion_prepared_parameters_valid(&no));
no=prep;no.wire_flags=4;assert(!d4r_native_motion_prepared_parameters_valid(&no));
no=prep;no.reserved=1;assert(!d4r_native_motion_prepared_parameters_valid(&no));
return 0;
}
'''
with tempfile.TemporaryDirectory() as tmp:
runner = Path(tmp) / "native-motion"
for compiler, language, standard in [("cc", "c", "c11"), ("c++", "c++", "c++17")]:
subprocess.run([compiler, "-x", language, "-std=" + standard, "-Wall", "-Wextra", "-Werror",
"-I", str(ROOT / "tools"), "-o", str(runner), "-"],
input=source, text=True, check=True, capture_output=True)
subprocess.run([str(runner)], check=True, timeout=5)
tests/test_native_motion_routing.py
+81
View File
@@ -0,0 +1,81 @@
"""Execute the shim's motion routing with native readiness and NGX failures."""
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
class NativeMotionRoutingTests(unittest.TestCase):
def test_independent_repair_and_ngx_motion(self):
source = (ROOT / 'tools/d4r_nvngx_shim.cpp').read_text()
begin = source.index(' CudaObject nativeMotion = 0;')
end = source.index(' // Ray Reconstruction: the auxiliary', begin)
block = source[begin:end]
begin = source.index(' const bool wireHistoryRequested=nativeMotion')
end = source.index(';', begin) + 1
wire = source[begin:end]
program = r'''
#include <atomic>
#include <cassert>
#include <cstdint>
#include <cstring>
using CudaObject = uint64_t;
static unsigned substitute=1, repair=1;
static bool available=true, matched=true;
static CudaObject selected=0;
static unsigned env_uint(const char* name, unsigned) {
return std::strcmp(name,"D4R_NATIVE_MV_SUBSTITUTE")==0 ? substitute : repair;
}
static void logf(const char*, ...) {}
struct Feature {
unsigned flags=0x43, cudaFlags=0x43, outWidth=2560, outHeight=1440;
void* cudaHandle=(void*)1;
};
struct Slot { bool nativeLive=true; };
struct Params { unsigned renderWidth=853,renderHeight=480; float mvScaleX=853,mvScaleY=480; };
static CudaObject compose_native_live(Feature&, Slot&, const Params&, unsigned, void*) { return available ? 42 : 0; }
static bool set_native_motion_flags(Feature& f, Params&, bool native, unsigned) {
if(matched) f.cudaFlags=native ? f.flags & ~2u : f.flags;
return matched;
}
static CudaObject prepare_dilated_motion(Feature&, Slot&, const Params&) { return 7; }
struct { struct { bool stabilizeThinWires=true; } cu; } g;
static void run(bool nativeLive, bool hasField, unsigned useNative, unsigned useRepair,
bool match, unsigned initialFlags, bool expectedOk,
CudaObject expectedMotion, bool expectedRepair) {
Feature state; auto* feature=&state; Slot slot; Params params;
slot.nativeLive=nativeLive; available=hasField; substitute=useNative;repair=useRepair;matched=match;
state.cudaFlags=initialFlags;
bool ok=true; unsigned frame=1; void* stages=nullptr;
''' + block + wire + r'''
selected=liveMotion ? liveMotion : dilatedMotion;
assert(ok==expectedOk);
assert(selected==expectedMotion);
assert(wireHistoryRequested==expectedRepair);
if(liveMotion) {
assert(params.mvScaleX==2560 && params.mvScaleY==1440);
assert(state.cudaFlags==0x41);
} else {
assert(params.mvScaleX==853 && params.mvScaleY==480);
if(ok) assert(state.cudaFlags==0x43);
}
}
int main() {
run(true,true,1,1,true,0x43,true,42,true); // old combined path
run(true,true,1,0,true,0x43,true,42,false); // native only
run(true,true,0,1,true,0x43,true,7,true); // game MVs + repair
run(true,true,0,0,true,0x43,true,7,false); // game MVs, no repair
run(false,false,1,1,true,0x43,true,7,false); // clean baseline
run(true,false,0,1,true,0x43,true,7,false); // unavailable native inputs
run(true,true,0,1,true,0x41,true,7,true); // restore low-res NGX flags
run(true,true,0,1,false,0x41,false,0,true); // failed restore must not evaluate
run(true,true,1,1,false,0x43,true,7,false); // preserve old raw-output fallback
}
'''
with tempfile.TemporaryDirectory() as temp:
runner = Path(temp) / 'routing'
subprocess.run(['c++', '-std=c++17', '-Wall', '-Wextra', '-Werror', '-x', 'c++', '-',
'-o', str(runner)], input=program, text=True, check=True, capture_output=True)
subprocess.run([str(runner)], check=True, timeout=5)
tests/test_optiscaler_packaging.py
+36 -3
View File
@@ -91,7 +91,7 @@ class OptiScalerConfigTests(unittest.TestCase):
inputs.mkdir()
(repo / 'scripts').mkdir(parents=True)
for name in ('package_release.sh', 'configure_optiscaler.py', 'check_glibc_compat.sh',
'check_redistributable.py', 'private_d3d12_core.py'):
'check_redistributable.py', 'private_d3d12_core.py', 'stage_engine_kit.sh'):
shutil.copy2(ROOT / 'scripts' / name, repo / 'scripts' / name)
shutil.copytree(ROOT / 'packaging', repo / 'packaging')
for name in ('LICENSE', 'NOTICE'):
@@ -113,6 +113,18 @@ class OptiScalerConfigTests(unittest.TestCase):
path.write_bytes(data)
return path
# Exercise the actual installer-kit stager with representative source files.
# No model builder, shader compiler or GPU fixture is executed here.
for name in ('engine/compile_k.py', 'engine/test_k.py', 'engine/test_m.py',
'engine/rr/test_layer.py', 'vulkan/ptx2glsl.py'):
destination = repo / name
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(ROOT / name, destination)
fixture(repo / 'engine/weights.bin', b'must not enter the model kit')
fixture(repo / 'engine/test_unrelated.py', b'must not enter the model kit')
fixture(repo / 'build/d3d12_dlss_harness.exe')
validator = fixture(inputs / 'glslang/bin/glslangValidator', b'packaging-only compiler fixture')
# A real minimal ELF lets the package's existing GLIBC checker run.
elf = inputs / 'library.so'
subprocess.run(['cc', '-shared', '-x', 'c', '-o', str(elf), '-'],
@@ -153,7 +165,8 @@ class OptiScalerConfigTests(unittest.TestCase):
D4R_ZLUDA_DIR=str(inputs / 'zluda'), D4R_VKD3D_DIR=str(inputs / 'vkd3d'),
D4R_ROCM_RUNTIME=str(inputs / 'rocm'), D4R_OPTISCALER_LICENSE=str(license_path),
D4R_ZLUDA_SRC=str(inputs / 'source'), D4R_VKD3D_SRC=str(inputs / 'source'),
D4R_BUNDLE_DLSS=str(dlss), D4R_BUNDLE_NGX=str(core), D4R_BUNDLE_TEX=str(inputs / 'tex'))
D4R_BUNDLE_DLSS=str(dlss), D4R_BUNDLE_NGX=str(core), D4R_BUNDLE_TEX=str(inputs / 'tex'),
D4R_GLSLANG=str(inputs / 'glslang'), D4R_GLSLANG_LICENSE=str(license_path))
version = (repo / 'packaging/VERSION').read_text().strip()
configs = []
# A compiled engine model: only d4r's shaders and marker may reach a zip.
@@ -186,12 +199,32 @@ class OptiScalerConfigTests(unittest.TestCase):
self.assertEqual(archive.read('d4r.asi'), archive.read('d4r/nvngx.dll'))
self.assertIn('d4r/d4r-setup.exe', archive.namelist())
self.assertEqual('d4r/install.sh' in archive.namelist(), bundled == 'installer')
self.assertFalse(any(n.endswith('.py') for n in archive.namelist()))
if bundled != 'installer':
self.assertFalse(any(n.endswith('.py') for n in archive.namelist()))
self.assertIn(b'ASI add-on', archive.read('d4r/README.txt'))
self.assertEqual({n.split('/')[0] for n in archive.namelist()}, {'d4r.asi', 'd4r'})
self.assertIn('d4r/d3d12.dll', archive.namelist())
self.assertIn('d4r/d4r12core.dll', archive.namelist())
self.assertNotIn('d4r/d3d12core.dll', archive.namelist())
kit = 'd4r/tools/models/'
self.assertEqual('d4r/tools/build-engine-models.sh' in archive.namelist(), bundled == 'installer')
if bundled == 'installer':
self.assertEqual(archive.read('d4r/tools/build-engine-models.sh'),
(ROOT / 'packaging/build_engine_models.sh').read_bytes())
for executable in ('d4r/tools/build-engine-models.sh', kit + 'bin/glslangValidator'):
self.assertTrue((archive.getinfo(executable).external_attr >> 16) & 0o111)
for source in ('engine/compile_k.py', 'engine/test_k.py', 'engine/test_m.py',
'engine/rr/test_layer.py', 'vulkan/ptx2glsl.py'):
self.assertEqual(archive.read(kit + source), (ROOT / source).read_bytes())
self.assertEqual(archive.read(kit + 'bin/glslangValidator'), validator.read_bytes())
self.assertEqual(archive.read('d4r/licenses/glslang-LICENSE.txt'), license_path.read_bytes())
self.assertIn(kit + 'bin/d3d12_dlss_harness.exe', archive.namelist())
self.assertFalse(any(n.endswith('.py') and not n.startswith(kit)
for n in archive.namelist()))
self.assertNotIn(kit + 'engine/weights.bin', archive.namelist())
self.assertNotIn(kit + 'engine/test_unrelated.py', archive.namelist())
else:
self.assertFalse(any(n.startswith(kit) for n in archive.namelist()))
self.assertIn('d4r/nvngx.dll', archive.namelist())
self.assertNotIn('nvngx.dll', archive.namelist())
self.assertNotIn('d4r/_nvngx.dll', archive.namelist())
tests/test_output_conversion_gpu.py
+21
View File
@@ -0,0 +1,21 @@
"""R11G11B10 storage conversion must match the original blit for all half inputs."""
import os
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT=Path(__file__).resolve().parents[1]
@unittest.skipUnless(os.environ.get('D4R_RUN_GPU_TESTS')=='1','opt-in GPU test')
class OutputConversionGpuTests(unittest.TestCase):
def test_fused_r11_matches_blit(self):
with tempfile.TemporaryDirectory() as temp:
runner=Path(temp)/'conversion'
subprocess.run(['c++','-std=c++17','-O2','-Wall','-Wextra','-Werror',
'-Wno-missing-field-initializers','-I'+str(ROOT/'tools'),
'-I'+str(ROOT/'build/vulkan-staging'),
str(ROOT/'tools/vulkan_output_conversion_probe.cpp'),'-lvulkan','-o',str(runner)],
check=True,capture_output=True,text=True)
subprocess.run([str(runner)],check=True,capture_output=True,text=True,timeout=30)
tests/test_prepared_wire_gpu.py
+83
View File
@@ -0,0 +1,83 @@
"""Prepared geometry keeps rendered output and readable temporal state exact."""
import os
from pathlib import Path
import random
import struct
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
@unittest.skipUnless(os.environ.get('D4R_RUN_GPU_TESTS') == '1', 'opt-in GPU test')
class PreparedWireGpuTests(unittest.TestCase):
def test_prepared_matches_reference_with_poisoned_invalid_history(self):
runner = os.environ['D4R_DUMP_RUNNER']
kernel = ROOT/'build/wine-nvcuda/generated'/(os.environ.get('D4R_GPU_ARCH', 'gfx1101')+'.hsaco')
w,h,rw,rh=65,67,21,22
pitch,dp,mp,cp,idp=w*8+16,w*4+12,w*4+20,rw*8+16,rw*4+12
pixels=w*h
rng=random.Random(746)
rgba=lambda: struct.pack('<4e',rng.uniform(.05,1),rng.uniform(.05,1),rng.uniform(.05,1),.5)
current=b''.join(b''.join(rgba() for _ in range(w))+b'\xa5'*16 for _ in range(h))
source=b''.join(struct.pack('<4e',.125,.125,.125,.75)*rw+b'\xa5'*16 for _ in range(rh))
previous=bytearray(b''.join(struct.pack('<4e',.125,.125,.125,.75)*w+b'\xa5'*16 for _ in range(h)))
ages=bytes(rng.choice((0,3,4,15,16,255)) for _ in range(pixels))
previous_depth=bytearray(struct.pack('<f',.5)*pixels)
for i,age in enumerate(ages):
if age==255:
y,x=divmod(i,w)
previous[y*pitch+x*8:y*pitch+x*8+8]=struct.pack('<4H',0x7e00,0x7e00,0x7e00,0x7e00)
previous_depth[i*4:i*4+4]=struct.pack('<f',float('nan'))
for flags in (0,1,2,3):
for jitter in ((0.,0.),(.25,-.25),(-1.,1.)):
for pattern in ('sky','flat','wire'):
with self.subTest(flags=flags,jitter=jitter,pattern=pattern), tempfile.TemporaryDirectory() as tmp:
d=Path(tmp);out=d/'out';out.mkdir()
z=[1. if pattern=='sky' else .5 if pattern=='flat' or x==32 else 1.
for y in range(h) for x in range(w)]
if pattern=='wire':z[10*w+31]=float('nan')
depth=b''.join(struct.pack('<'+'f'*w,*z[y*w:(y+1)*w])+b'\xa5'*12 for y in range(h))
arrays=[current,source,depth,b''.join(bytes(w*4)+b'\xa5'*20 for _ in range(h)),
bytes(previous),ages,bytes(previous_depth),b'\xa5'*(pitch*h),
b'\xa5'*(pixels+64),b'\xa5'*(pixels*4+64),
b''.join(struct.pack('<f',.5)*rw+b'\xa5'*12 for _ in range(rh)),
b'\xa5'*(pitch*h),b'\xa5'*(pixels+64),
b'\xa5'*(((w+31)//32)*h*4+64),b'\xa5'*(pixels+64),
b'\xa5'*(mp*h),bytes(1136),bytes(64)]
addresses=[(i+1)*0x100000000 for i in range(len(arrays))]
def reset():
for i,data in enumerate(arrays):(d/f'alloc-{i}.bin').write_bytes(data)
def launch(name,args,pointers,grid):
(d/'args.bin').write_bytes(args)
manifest=['kernel '+name,'launch '+' '.join(map(str,(*grid,1,256,1,1,0))),f'args {len(args)}']
for i,data in enumerate(arrays):manifest.append(f'alloc {i} {hex(addresses[i])} {len(data)}')
manifest += [f'pointer {offset} {allocation} 0' for offset,allocation in pointers]
(d/'manifest.txt').write_text('\n'.join(manifest)+'\n')
subprocess.run([runner,str(kernel),name,str(d),str(out),'1'],check=True,capture_output=True)
for i in range(len(arrays)):(d/f'alloc-{i}.bin').write_bytes((out/f'alloc-{i}.bin').read_bytes())
args=struct.pack('<17Q5I2fI',*addresses[:12],cp,dp,mp,pitch,idp,w,h,rw,rh,1,*jitter,flags)
base_pointers=[(i*8,i) for i in range(12)]
reset();launch('d4r_wire_history',args,base_pointers,((w+15)//16,(h+15)//16))
expected={i:(out/f'alloc-{i}.bin').read_bytes() for i in (7,8,9,11)}
reset()
native=struct.pack('<8Q4I2f2Q2I',addresses[2],addresses[3],addresses[16],addresses[17],addresses[15],
dp,mp,mp,w,h,rw,rh,*jitter,addresses[13],addresses[14],flags,0)
native_pointers=[(0,2),(8,3),(16,16),(24,17),(32,15),(88,13),(96,14)]
launch('d4r_native_motion_prepare',native,native_pointers,((w+31)//32,(h+31)//32))
prepared=args+struct.pack('<2Q',addresses[12],addresses[14])
launch('d4r_wire_history_preclassified',prepared,base_pointers+[(168,12),(176,14)],((w+15)//16,(h+15)//16))
launch('d4r_wire_history_apply',prepared[:176],base_pointers+[(168,12)],((w+15)//16,(h+15)//16))
actual={i:(out/f'alloc-{i}.bin').read_bytes() for i in range(len(arrays))}
self.assertEqual(actual[0],expected[7])
self.assertEqual(actual[8],expected[8])
for i in (1,2,3,4,5,6,10,16,17):self.assertEqual(actual[i],arrays[i])
for i in range(pixels):
if expected[8][i]!=255:
y,x=divmod(i,w);offset=y*pitch+x*8
self.assertEqual(actual[11][offset:offset+8],expected[11][offset:offset+8])
self.assertEqual(actual[9][i*4:i*4+4],expected[9][i*4:i*4+4])
for y in range(h):self.assertEqual(actual[11][y*pitch+w*8:(y+1)*pitch],b'\xa5'*16)
for i,start in ((8,pixels),(9,pixels*4),(12,pixels),(13,((w+31)//32)*h*4),(14,pixels)):
self.assertEqual(actual[i][start:],b'\xa5'*64)
tests/test_profile_summary.py
+48
View File
@@ -0,0 +1,48 @@
"""Keep startup, feature instances and resolution changes out of shared medians."""
import importlib.util
from pathlib import Path
import unittest
ROOT = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location('profile_summary', ROOT/'scripts/summarize_d4r_profile.py')
summary = importlib.util.module_from_spec(spec)
spec.loader.exec_module(summary)
class ProfileSummaryTests(unittest.TestCase):
def test_warm_groups_and_unavailable_timings(self):
def sample(frame, render='853x480', reset=0, gpu='2.2'):
return (f'D4R_PROFILE frame={frame} render={render} output=2560x1440 '
f'preset=11 quality=3 flags=0x43 input_wait=cpu reset={reset} '
f'vk_total=2.8 dlss_gpu={gpu} native_motion_gpu=-1 wire_repair_gpu=-1')
create='NVSDK_NGX_D3D12_CreateFeature(feature=1)'
log='\n'.join((create, sample(1), sample(300, reset=1), sample(600),
sample(900, gpu='nan'), sample(1200, render='1706x960'),
create, sample(300)))
rows=summary.summarize(log)
self.assertEqual(len(rows), 3)
self.assertEqual([r['samples'] for r in rows], [2, 1, 1])
self.assertEqual([r['feature_instance'] for r in rows], [1, 1, 2])
self.assertEqual(rows[0]['timings_ms']['dlss_gpu']['n'], 1)
self.assertIsNone(rows[0]['timings_ms']['wire_repair_gpu'])
self.assertEqual(rows[0]['timings_ms']['vk_total']['median'], 2.8)
def test_routing_changes_get_separate_groups(self):
base = ('D4R_PROFILE frame=300 render=853x480 output=2560x1440 '
'preset=11 quality=3 flags=0x43 input_wait=cpu reset=0 '
'feature=29184 native_field=1 wire_requested=1 gpu_completion=0 ')
rows = summary.summarize(base+'native_substitute=0\n'+base+'native_substitute=1')
self.assertEqual(len(rows), 2)
self.assertEqual([r['native_substitute'] for r in rows], ['0', '1'])
def test_scene_window_and_actual_output_mode(self):
base = ('D4R_PROFILE frame=600 render=853x480 output=2560x1440 '
'preset=11 quality=3 flags=0x43 input_wait=cpu reset=0 ')
log = ('[ 10.000] '+base+'vk_total=8 vk_output_mode=transfer\n'
'[ 20.000] '+base+'vk_total=3 vk_output_mode=compute\n'
'[ 21.000] '+base+'vk_total=4 vk_output_mode=transfer\n'
'[ 30.000] '+base+'vk_total=9 vk_output_mode=compute')
rows = summary.summarize(log, since_clock=19, until_clock=22)
self.assertEqual(len(rows), 2)
self.assertEqual([r['vk_output_mode'] for r in rows], ['compute', 'transfer'])
self.assertEqual([r['timings_ms']['vk_total']['median'] for r in rows], [3, 4])
tests/test_ray_reconstruction.py
+61 -1
View File
@@ -7,7 +7,8 @@ import unittest
ROOT = Path(__file__).resolve().parents[1]
CONFIG = ROOT / "scripts" / "d4r_config.py"
RR_VARIABLES = ("D4R_RR_ENABLE", "D4R_RR_PRESET")
RR_VARIABLES = ("D4R_RR_ENABLE", "D4R_RR_PRESET", "D4R_RR_BACKEND", "D4R_FSR_RR_DLL", "D4R_FSR_RR_STABILITY_BIAS",
"D4R_FSR_RR_DETAIL")
def run_config(text, **overrides):
@@ -68,5 +69,64 @@ class RayReconstructionConfigTests(unittest.TestCase):
self.assertNotIn("D4R_RR_PRESET", decided)
class RayReconstructionBackendTests(unittest.TestCase):
"""[RayReconstruction] Backend picks the denoiser; it never changes the DLSS presets."""
def test_default_leaves_dlss_in_place(self):
code, decided = run_config("[RayReconstruction]\nEnable = true\n")
self.assertEqual(code, 0)
self.assertNotIn("D4R_RR_BACKEND", decided) # the shim's default is dlss
def test_backends_are_exported_and_independent_of_presets(self):
for backend in ("dlss", "fsr", "FSR"):
with self.subTest(backend=backend):
code, decided = run_config(f"[RayReconstruction]\nBackend = {backend}\nModel = E\n[DLSS]\nModel = K\n")
self.assertEqual(code, 0)
self.assertEqual(decided["D4R_RR_BACKEND"], backend.lower())
self.assertEqual(decided["D4R_RR_PRESET"], "5")
self.assertEqual(decided["D4R_DLSS_PRESET"], "11")
def test_unknown_backend_is_rejected(self):
for value in ("amd", "fsr4", "native"):
with self.subTest(value=value):
code, decided = run_config(f"[RayReconstruction]\nBackend = {value}\n")
self.assertNotEqual(code, 0)
self.assertNotIn("D4R_RR_BACKEND", decided)
def test_denoiser_path_and_environment_precedence(self):
code, decided = run_config("[RayReconstruction]\nBackend = fsr\nFsrDenoiser = /opt/amd/denoiser.dll\n")
self.assertEqual(code, 0)
self.assertEqual(decided["D4R_FSR_RR_DLL"], "/opt/amd/denoiser.dll")
code, decided = run_config("[RayReconstruction]\nBackend = fsr\n", D4R_RR_BACKEND="dlss")
self.assertEqual(code, 0)
self.assertNotIn("D4R_RR_BACKEND", decided)
def test_templates_keep_dlss_as_the_default(self):
import configparser
for name in ("config/d4r.ini.default", "packaging/d4r.ini"):
ini = configparser.ConfigParser(inline_comment_prefixes=(";",))
ini.read(ROOT / name)
self.assertEqual(ini.get("RayReconstruction", "Backend"), "dlss", name)
self.assertEqual(ini.get("RayReconstruction", "FsrDenoiser"), "auto", name)
self.assertEqual(ini.get("RayReconstruction", "FsrStabilityBias"), "auto", name)
self.assertEqual(ini.get("RayReconstruction", "FsrDetail"), "0", name)
def test_fsr_tuning_keys(self):
code, decided = run_config("[RayReconstruction]\nBackend = fsr\nFsrStabilityBias = auto\nFsrDetail = 0.25\n")
self.assertEqual(code, 0)
self.assertNotIn("D4R_FSR_RR_STABILITY_BIAS", decided) # auto: AMD's own default
self.assertEqual(decided["D4R_FSR_RR_DETAIL"], "0.25")
code, decided = run_config("[RayReconstruction]\nFsrStabilityBias = 0\n")
self.assertEqual(code, 0)
self.assertEqual(decided["D4R_FSR_RR_STABILITY_BIAS"], "0")
for text in ("FsrStabilityBias = -1", "FsrStabilityBias = low", "FsrDetail = 1.5", "FsrDetail = x"):
with self.subTest(text=text):
code, _ = run_config(f"[RayReconstruction]\n{text}\n")
self.assertNotEqual(code, 0)
code, decided = run_config("[RayReconstruction]\nFsrStabilityBias = 0\n", D4R_FSR_RR_STABILITY_BIAS="1")
self.assertEqual(code, 0)
self.assertNotIn("D4R_FSR_RR_STABILITY_BIAS", decided)
if __name__ == "__main__":
unittest.main()
tests/test_rr_camera.py
+46
View File
@@ -0,0 +1,46 @@
"""CPU camera transforms: inversion and relative camera motion before RR dispatch."""
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
class RrCameraTests(unittest.TestCase):
def test_inverse_and_relative_camera_translation(self):
source = r'''
#include "engine/rr_camera.h"
#include <cassert>
#include <cmath>
using namespace d4r::rr_camera;
int main() {
const Matrix identity{1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,0,1};
// A dense, asymmetric matrix catches row/column and cofactor-sign mistakes.
const Matrix dense{2,1,3,4, 0,3,2,1, 1,0,4,2, 2,3,0,5};
const Matrix product = multiply(dense, inverse(dense));
for (int i = 0; i < 16; ++i) assert(std::abs(product[i] - identity[i]) < 2e-6f);
Matrix current = identity, previous = identity;
current[12] = 3; previous[12] = 2;
const auto camera = transforms(current, identity, previous, identity, 4, 2);
Matrix expectedPrevious = identity; expectedPrevious[12] = -1;
const Matrix pixels{2,0,0,0, 0,-1,0,0, 0,0,1,0, 2,1,0,1};
const Matrix expectedPixels = multiply(expectedPrevious, pixels);
for (int i = 0; i < 16; ++i) {
assert(camera.previousClip[i] == expectedPrevious[i]);
assert(camera.inverseClip[i] == identity[i]);
assert(camera.previousPixels[i] == expectedPixels[i]);
}
const Matrix inversePixelProduct = multiply(pixels, camera.inversePixels);
for (int i = 0; i < 16; ++i) assert(inversePixelProduct[i] == identity[i]);
// Reset is represented by passing the current camera as the previous camera.
const auto reset = transforms(current, identity, current, identity, 4, 2);
for (int i = 0; i < 16; ++i) assert(reset.previousClip[i] == identity[i]);
}
'''
with tempfile.TemporaryDirectory() as temporary:
runner = Path(temporary) / 'camera'
subprocess.run(['c++', '-O2', '-std=c++17', '-Wall', '-Wextra', '-Werror',
'-I', str(ROOT), '-x', 'c++', '-', '-o', str(runner)],
input=source, text=True, check=True, capture_output=True)
subprocess.run([str(runner)], check=True, timeout=10)
tests/test_shim_config.py
+34 -4
View File
@@ -1,4 +1,4 @@
"""Engine opt-in, preset-selection behavior and polling configuration precedence."""
"""Engine defaults, preset-selection behavior and polling configuration precedence."""
import configparser
import os
from pathlib import Path
@@ -10,17 +10,47 @@ ROOT = Path(__file__).resolve().parents[1]
class ShimConfigTests(unittest.TestCase):
def test_engine_is_opt_in_and_configured_by_ini(self):
def test_profile_log_interval_validation_and_environment_precedence(self):
with tempfile.TemporaryDirectory() as temporary:
ini = Path(temporary) / 'd4r.ini'
env = {key: value for key, value in os.environ.items() if not key.startswith('D4R_')}
cmd = ['python3', str(ROOT / 'scripts/d4r_config.py'), '--config', str(ini)]
for value, expected in ((None, None), ('auto', None), ('1', '1'), ('60', '60'),
('060', '60'), ('120', '120'), ('4294967295', '4294967295')):
ini.write_text('[Debug]\nProfile = true\n' +
(f'ProfileLogEvery = {value}\n' if value is not None else ''))
result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=True)
self.assertIn('export D4R_PROFILE=1\n', result.stdout)
if expected is None:
self.assertNotIn('export D4R_PROFILE_LOG_EVERY=', result.stdout)
else:
self.assertIn(f'export D4R_PROFILE_LOG_EVERY={expected}\n', result.stdout)
ini.write_text('[Debug]\nProfileLogEvery = 60\n')
result = subprocess.run(cmd, env=dict(env, D4R_PROFILE_LOG_EVERY='120'),
text=True, capture_output=True, check=True)
self.assertNotIn('export D4R_PROFILE_LOG_EVERY=', result.stdout)
for value in ('0', '-1', '0.5', 'invalid', '4294967296'):
ini.write_text(f'[Debug]\nProfileLogEvery = {value}\n')
result = subprocess.run(cmd, env=env, text=True, capture_output=True)
self.assertEqual(result.returncode, 2, value)
for name in ('packaging/d4r.ini', 'config/d4r.ini.default'):
ini = configparser.ConfigParser(inline_comment_prefixes=(';',))
ini.read(ROOT / name)
self.assertFalse(ini.getboolean('Engine', 'Enabled'), name)
self.assertEqual(ini.getint('Debug', 'ProfileLogEvery'), 60, name)
def test_engine_defaults_and_ini_overrides(self):
for name, expected in (('packaging/d4r.ini', 'auto'), ('config/d4r.ini.default', 'false')):
ini = configparser.ConfigParser(inline_comment_prefixes=(';',))
ini.read(ROOT / name)
self.assertEqual(ini.get('Engine', 'Enabled'), expected, name)
self.assertEqual(ini.get('Engine', 'ModelDir'), 'auto', name)
with tempfile.TemporaryDirectory() as temporary:
ini = Path(temporary) / 'd4r.ini'
env = {key: value for key, value in os.environ.items() if not key.startswith('D4R_')}
cmd = ['python3', str(ROOT / 'scripts/d4r_config.py'), '--config', str(ini)]
for text, enabled, model in (('', '0', None), ('[Engine]\nEnabled = true\n', '1', None),
for text, enabled, model in (('', '0', None), ('[Engine]\nEnabled = auto\n', '0', None),
('[Engine]\nEnabled = false\n', '0', None),
('[Engine]\nEnabled = true\n', '1', None),
('[Engine]\nEnabled = true\nModelDir = /models/k\n', '1', '/models/k')):
ini.write_text(text)
result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=True)
tests/test_vram_profile_timing.py
+42
View File
@@ -0,0 +1,42 @@
"""Conversion labels must belong to the older Vulkan query slot, not this frame."""
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT=Path(__file__).resolve().parents[1]
class VramProfileTimingTests(unittest.TestCase):
def test_older_slot_mode_labels(self):
source=(ROOT/'tools/d4r_nvngx_shim.cpp').read_text()
start=source.index('static void read_profile_timestamps(Feature& feature, uint32_t frame, FrameTiming& timing)\n{')
end=source.index('\n}\n',start)+3
program=r'''
#include <cassert>
#include <cstdint>
constexpr unsigned kOutputSlots=4;
struct FrameTiming {double vkInput=-1,vkDlssWait=-1,vkOutput=-1,vkTotal=-1;int vkOutputFused=-1,vkInputFused=-1;};
struct Feature {uint64_t values[16]={};volatile uint64_t* timestampValues=values;double timestampTicksPerMs=10;
uint8_t timestampOutputFused[4]={},timestampInputFused[4]={};bool recordOutputFused=true,recordInputFused=true;};
''' + source[start:end] + r'''
int main() {
Feature f;FrameTiming old;
f.values[0]=10;f.values[1]=20;f.values[2]=50;f.values[3]=60;
f.timestampOutputFused[0]=0;f.timestampInputFused[0]=1;
read_profile_timestamps(f,300,old); // frame 296 used transfer output, despite current mode switching on
assert(old.vkOutputFused==0 && old.vkInputFused==1 && old.vkTotal==5);
assert(!f.recordOutputFused && !f.recordInputFused && f.values[0]==0);
f.values[0]=70;f.values[1]=80;f.values[2]=110;f.values[3]=120;
f.timestampOutputFused[0]=1;f.timestampInputFused[0]=0;FrameTiming next;
read_profile_timestamps(f,304,next);
assert(next.vkOutputFused==1 && next.vkInputFused==0);
FrameTiming unavailable;read_profile_timestamps(f,308,unavailable);
assert(unavailable.vkOutputFused==-1 && unavailable.vkInputFused==-1 && unavailable.vkTotal==-1);
}
'''
with tempfile.TemporaryDirectory() as temp:
runner=Path(temp)/'timing'
subprocess.run(['c++','-std=c++17','-Wall','-Wextra','-Werror','-x','c++','-',
'-o',str(runner)],input=program,text=True,check=True,capture_output=True)
subprocess.run([str(runner)],check=True,timeout=5)
tests/test_wire_background_gpu.py
+64
View File
@@ -0,0 +1,64 @@
"""Fractional wire coverage must preserve a different foreground surface."""
import os
from pathlib import Path
import struct
import subprocess
import tempfile
import unittest
ROOT=Path(__file__).resolve().parents[1]
@unittest.skipUnless(os.environ.get('D4R_RUN_GPU_TESTS')=='1','opt-in GPU test')
class WireBackgroundGpuTests(unittest.TestCase):
def test_sky_wire_does_not_spill_onto_other_geometry(self):
runner=os.environ['D4R_DUMP_RUNNER']
kernel=Path(os.environ.get('D4R_WIRE_TEST_KERNEL',str(ROOT/'build/wine-nvcuda/generated'/'gfx1101.hsaco')))
w,h=65,67
pitch,dp,mp=w*8+16,w*4+12,w*4+20
pixels=w*h
rgba=struct.pack('<4e',.5,.5,.5,.5)
observed=struct.pack('<4e',.125,.125,.125,.75)
current=b''.join(rgba*w+b'\xa5'*16 for _ in range(h))
color=b''.join(observed*w+b'\xa5'*16 for _ in range(h))
depth=b''.join(struct.pack('<'+'f'*w,*[
.5 if y==32 else .9 if x<=31 and y<=31 else 1. for x in range(w)])+b'\xa5'*12 for y in range(h))
motion=b''.join(bytes(w*4)+b'\xa5'*20 for _ in range(h))
input_depth=b''.join(struct.pack('<f',.5)*w+b'\xa5'*12 for _ in range(h))
for sparse in (False,True):
with self.subTest(sparse=sparse), tempfile.TemporaryDirectory() as temp:
directory=Path(temp);out=directory/'out';out.mkdir()
arrays=[current,color,depth,motion,color,b'\xff'*pixels,struct.pack('<f',1.)*pixels,
b'\xa5'*(pitch*h),b'\xa5'*(pixels+64),b'\xa5'*(pixels*4+64),
input_depth,b'\xa5'*(pitch*h)]
if sparse:arrays.append(b'\xa5'*(pixels+64))
addresses=[(i+1)*0x100000000 for i in range(len(arrays))]
args=struct.pack('<17Q5I2fI',*addresses[:12],pitch,dp,mp,pitch,dp,w,h,w,h,0,.25,.25,3)
if sparse:args+=struct.pack('<Q',addresses[12])
(directory/'args.bin').write_bytes(args)
manifest=['kernel '+('d4r_wire_history_sparse' if sparse else 'd4r_wire_history'),
'launch 5 5 1 256 1 1 0',f'args {len(args)}']
for i,data in enumerate(arrays):
(directory/f'alloc-{i}.bin').write_bytes(data)
manifest += [f'alloc {i} {hex(addresses[i])} {len(data)}',f'pointer {168 if i==12 else i*8} {i} 0']
(directory/'manifest.txt').write_text('\n'.join(manifest)+'\n')
name='d4r_wire_history_sparse' if sparse else 'd4r_wire_history'
subprocess.run([runner,str(kernel),name,str(directory),str(out),'1'],check=True,capture_output=True)
if sparse:
for i in range(len(arrays)):(directory/f'alloc-{i}.bin').write_bytes((out/f'alloc-{i}.bin').read_bytes())
manifest[0]='kernel d4r_wire_history_apply'
(directory/'manifest.txt').write_text('\n'.join(manifest)+'\n')
subprocess.run([runner,str(kernel),'d4r_wire_history_apply',str(directory),str(out),'1'],check=True,capture_output=True)
result=(out/f'alloc-{0 if sparse else 7}.bin').read_bytes()
# The .0625 wire corner overlaps a different, thick foreground object.
self.assertEqual(result[31*pitch+31*8:31*pitch+32*8],rgba)
self.assertEqual((out/'alloc-8.bin').read_bytes()[31*w+31],255)
# The neighboring actual wire remains repairable; alpha stays untouched.
wire=result[32*pitch+32*8:32*pitch+33*8]
self.assertNotEqual(wire,rgba)
self.assertEqual(wire[6:],rgba[6:])
for i in ((1,2,3,4,5,6,10) if sparse else (0,1,2,3,4,5,6,10)):
self.assertEqual((out/f'alloc-{i}.bin').read_bytes(),arrays[i])
for y in range(h):self.assertEqual(result[y*pitch+w*8:(y+1)*pitch],b'\xa5'*16)
for i,end in ((8,pixels),(9,pixels*4)):
self.assertEqual((out/f'alloc-{i}.bin').read_bytes()[end:],b'\xa5'*64)
tests/test_wire_history.py
+54
View File
@@ -0,0 +1,54 @@
"""Sparse wire publication rejects aliased scratch, inputs and overflowing ranges."""
from pathlib import Path
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
class WireHistoryTests(unittest.TestCase):
def test_sparse_buffer_contract(self):
source = r'''
#include "d4r_wire_history.h"
#include <assert.h>
#include <string.h>
int main(void) {
D4rWireHistorySparseParams p; memset(&p,0,sizeof(p));
D4rWireHistoryParams* b=&p.base;
b->width=65;b->height=67;b->render_width=21;b->render_height=22;
b->current=0x10000000;b->color=0x20000000;b->native_depth=0x30000000;
b->motion=0x40000000;b->previous=0x50000000;b->previous_age=0x60000000;
b->previous_depth=0x70000000;b->output=0x80000000;b->output_age=0x90000000;
b->output_depth=0xa0000000;b->input_depth=0xb0000000;b->output_history=0xc0000000;
b->color_pitch=21*8;b->depth_pitch=b->motion_pitch=65*4;
b->output_pitch=65*8;b->input_depth_pitch=21*4;p.patch_mask=0xd0000000;
assert(sizeof(p)==176 && d4r_wire_history_sparse_parameters_valid(&p));
assert(!d4r_wire_history_sparse_parameters_valid(0));
D4rWireHistorySparseParams q=p;q.patch_mask=0;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.patch_mask=UINT64_MAX-16;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.patch_mask=q.base.current+8;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.patch_mask=q.base.previous_age+1;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.base.current=q.base.color;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.base.output=q.base.native_depth+4;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.base.current=q.base.output_history+8;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.base.output_pitch=UINT64_MAX-7;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.base.current=UINT64_MAX-1024;assert(!d4r_wire_history_sparse_parameters_valid(&q));
q=p;q.patch_mask=q.base.current+66*q.base.output_pitch+65*8;
assert(d4r_wire_history_sparse_parameters_valid(&q));
D4rWireHistoryPreparedParams prepared={p,0xe0000000};
assert(sizeof(prepared)==184 && d4r_wire_history_prepared_parameters_valid(&prepared));
D4rWireHistoryPreparedParams no=prepared;no.axes=0;assert(!d4r_wire_history_prepared_parameters_valid(&no));
no=prepared;no.axes=no.sparse.base.current;assert(!d4r_wire_history_prepared_parameters_valid(&no));
no=prepared;no.axes=no.sparse.patch_mask;assert(!d4r_wire_history_prepared_parameters_valid(&no));
no=prepared;no.axes=UINT64_MAX-16;assert(!d4r_wire_history_prepared_parameters_valid(&no));
return 0;
}
'''
with tempfile.TemporaryDirectory() as tmp:
runner = Path(tmp)/'wire-contract'
for compiler, language, standard in (('cc', 'c', 'c11'), ('c++', 'c++', 'c++17')):
subprocess.run([compiler, '-x', language, '-std='+standard, '-Wall', '-Wextra', '-Werror',
'-I', str(ROOT/'tools'), '-o', str(runner), '-'],
input=source, text=True, check=True, capture_output=True)
subprocess.run([str(runner)], check=True)
tests/test_wire_history_gpu.py
+88
View File
@@ -0,0 +1,88 @@
"""Opt-in GPU checks for wire color, history eligibility, padding and immutable inputs."""
import os
from pathlib import Path
import struct
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
@unittest.skipUnless(os.environ.get('D4R_RUN_GPU_TESTS') == '1', 'opt-in GPU test')
class WireHistoryGpuTests(unittest.TestCase):
def test_wire_source_history_reset_and_expiry(self):
runner = os.environ['D4R_DUMP_RUNNER']
kernel = Path(os.environ.get('D4R_WIRE_TEST_KERNEL', str(ROOT / 'build/wine-nvcuda/generated' /
(os.environ.get('D4R_GPU_ARCH', 'gfx1101') + '.hsaco'))))
w, h = 65, 67 # partial workgroups on both axes
pitch, dp, mp = w*8+16, w*4+12, w*4+20
pixels = w*h
rgba = struct.pack('<4e', .5, .5, .5, .5)
observed = struct.pack('<4e', .125, .125, .125, .75)
repaired = struct.pack('<4e', .125, .125, .125, .5)
current = b''.join(rgba*w + b'\xa5'*16 for _ in range(h))
color = b''.join(observed*w + b'\xa5'*16 for _ in range(h))
native = b''.join(struct.pack('<'+'f'*w, *[.5 if x == 32 else 1. for x in range(w)]) +
b'\xa5'*12 for _ in range(h))
motion = b''.join(b'\0'*(w*4) + b'\xa5'*20 for _ in range(h))
for sky_only in (False, True):
for conservative in (False, True):
for sparse in (False, True):
for mode in ('source', 'history', 'reset', 'expired', 'disoccluded', 'missing', 'neighbor', 'far', 'backdrop'):
for jitter in (0, -1, 1):
with self.subTest(mode=mode, jitter=jitter, conservative=conservative, sparse=sparse, sky_only=sky_only), tempfile.TemporaryDirectory() as tmp:
directory = Path(tmp); output = directory/'out'; output.mkdir()
source = mode in ('source', 'backdrop')
history = mode in ('history', 'expired', 'disoccluded', 'neighbor')
z = .999988 if mode == 'far' else .5
near = b''.join(struct.pack('<'+'f'*w, *[z if x == 32 else .9 if mode == 'backdrop' else 1. for x in range(w)]) + b'\xa5'*12 for _ in range(h))
previous = color
if mode == 'neighbor':
previous = b''.join(b''.join(observed if x == 32-jitter else struct.pack('<4e', .01, .01, .01, .75) for x in range(w)) + b'\xa5'*16 for _ in range(h))
input_depth = b''.join(struct.pack('<f', z+1e-6 if mode == 'far' else .5 if source else 1.)*w + b'\xa5'*12 for _ in range(h))
arrays = [current, color, near, motion, previous,
bytes([(4 if conservative else 16) if mode == 'expired' else (3 if conservative else 15)])*pixels,
struct.pack('<f', 0. if mode == 'disoccluded' else .5)*pixels,
b'\xa5'*(pitch*h), b'\xa5'*(pixels+64), b'\xa5'*(pixels*4+64),
input_depth, b'\xa5'*(pitch*h)]
if sparse: arrays.append(b'\xa5'*(pixels+64))
addresses = [(i+1)*0x100000000 for i in range(len(arrays))]
args = struct.pack('<17Q5I2fI', *addresses[:12], pitch, dp, mp, pitch, dp,
w, h, w, h, int(history), jitter, -jitter, int(conservative) | (2 if sky_only else 0))
if sparse: args += struct.pack('<Q', addresses[12])
(directory/'args.bin').write_bytes(args)
manifest = ['kernel '+('d4r_wire_history_sparse' if sparse else 'd4r_wire_history'), 'launch 5 5 1 256 1 1 0', f'args {len(args)}']
for i, data in enumerate(arrays):
(directory/f'alloc-{i}.bin').write_bytes(data)
manifest += [f'alloc {i} {hex(addresses[i])} {len(data)}', f'pointer {168 if i == 12 else i*8} {i} 0']
(directory/'manifest.txt').write_text('\n'.join(manifest)+'\n')
if sparse:
subprocess.run([runner, str(kernel), 'd4r_wire_history_sparse', str(directory), str(output), '1'], check=True, capture_output=True)
for i in range(len(arrays)):
(directory/f'alloc-{i}.bin').write_bytes((output/f'alloc-{i}.bin').read_bytes())
manifest[0]='kernel d4r_wire_history_apply'
(directory/'manifest.txt').write_text('\n'.join(manifest)+'\n')
subprocess.run([runner, str(kernel), 'd4r_wire_history_apply', str(directory), str(output), '1'], check=True, capture_output=True)
else:
subprocess.run([runner, str(kernel), 'd4r_wire_history', str(directory), str(output), '1'], check=True, capture_output=True)
for i in ((1, 2, 3, 4, 5, 6, 10) if sparse else (0, 1, 2, 3, 4, 5, 6, 10)):
self.assertEqual((output/f'alloc-{i}.bin').read_bytes(), arrays[i])
expected = bytearray(current)
cache = bytearray(current)
age = bytearray(b'\xff'*pixels + b'\xa5'*64)
depth = bytearray(struct.pack('<f', 1.)*pixels + b'\xa5'*64)
for y in range(16, h-16):
if mode == 'backdrop' and sky_only: continue
x = 32-jitter
offset, index = y*pitch+x*8, y*w+x
depth[index*4:index*4+4] = struct.pack('<f', z)
if mode in ('source', 'history', 'neighbor', 'backdrop') or (mode == 'far' and not conservative):
value = .01 if mode == 'neighbor' and not conservative else .125
expected[offset:offset+8] = struct.pack('<4e', value, value, value, .5)
cache[offset:offset+8] = struct.pack('<4e', value, value, value, .75)
age[index] = 0 if source or mode == 'far' else 4 if conservative else 16
self.assertEqual((output/f'alloc-{0 if sparse else 7}.bin').read_bytes(), expected)
self.assertEqual((output/'alloc-8.bin').read_bytes(), age)
self.assertEqual((output/'alloc-9.bin').read_bytes(), depth)
self.assertEqual((output/'alloc-11.bin').read_bytes(), cache)
third_party/fidelityfx/LICENSE.txt
Vendored
+20
View File
@@ -0,0 +1,20 @@
These headers are copied unmodified from the AMD FidelityFX SDK
(https://github.com/GPUOpen-LibrariesAndSDKs/FidelityFX-SDK), commit
60f4ea81909200d8542eca14dccb2628b763a9a3 (SDK 2.3.0, FSR Ray Regeneration 1.2.0):
api/include/ffx_api.h
api/include/ffx_api_types.h
denoisers/include/ffx_denoiser.h
The SDK license (Kits/FidelityFX/docs/license.md) lists these files as exceptions
that are subject to the following MIT license. The AMD runtime DLLs
(amd_fidelityfx_*_dx12.dll) are NOT part of this repository and are governed by
AMD's binary-only terms.
Copyright (C) Advanced Micro Devices, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
third_party/fidelityfx/api/include/ffx_api.h
Vendored
+213
View File
@@ -0,0 +1,213 @@
// This file is part of the FidelityFX SDK.
//
// Copyright (C) 2026 Advanced Micro Devices, Inc.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files(the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and /or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions :
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
#pragma once
#if defined(__cplusplus)
extern "C" {
#endif // #if defined(__cplusplus)
#define FFX_API_ENTRY __declspec(dllexport)
#include <stdint.h>
enum FfxApiReturnCodes
{
FFX_API_RETURN_OK = 0, ///< The oparation was successful.
FFX_API_RETURN_ERROR = 1, ///< An error occurred that is not further specified.
FFX_API_RETURN_ERROR_UNKNOWN_DESCTYPE = 2, ///< The structure type given was not recognized for the function or context with which it was used. This is likely a programming error.
FFX_API_RETURN_ERROR_RUNTIME_ERROR = 3, ///< The underlying runtime (e.g. D3D12, Vulkan) or effect returned an error code.
FFX_API_RETURN_NO_PROVIDER = 4, ///< No provider was found for the given structure type. This is likely a programming error.
FFX_API_RETURN_ERROR_MEMORY = 5, ///< A memory allocation failed.
FFX_API_RETURN_ERROR_PARAMETER = 6, ///< A parameter was invalid, e.g. a null pointer, empty resource or out-of-bounds enum value.
FFX_API_RETURN_PROVIDER_NO_SUPPORT_NEW_DESCTYPE = 7, ///< The structure type given is new and not supported in the old provider. This is likely fixed with driver upgrade or effect DLL upgrade.
};
typedef void* ffxContext;
typedef uint32_t ffxReturnCode_t;
#define FFX_API_EFFECT_MASK 0x00ff0000u
#define FFX_API_BACKEND_MASK 0xff000000u
#define FFX_API_EFFECT_ID_GENERAL 0x00000000u
// Base Descriptor types
typedef uint64_t ffxStructType_t;
typedef struct ffxApiHeader
{
ffxStructType_t type; ///< The structure type. Must always be set to the corresponding value for any structure (found nearby with a similar name).
struct ffxApiHeader* pNext; ///< Pointer to next structure, used for optional parameters and extensions. Can be null.
} ffxApiHeader;
typedef ffxApiHeader ffxCreateContextDescHeader;
typedef ffxApiHeader ffxConfigureDescHeader;
typedef ffxApiHeader ffxQueryDescHeader;
typedef ffxApiHeader ffxDispatchDescHeader;
// Extensions for global debug
#define FFX_API_CONFIGURE_GLOBALDEBUG_LEVEL_SILENCE 0x0000000u
#define FFX_API_CONFIGURE_GLOBALDEBUG_LEVEL_ERRORS 0x0000001u
#define FFX_API_CONFIGURE_GLOBALDEBUG_LEVEL_WARNINGS 0x0000002u
#define FFX_API_CONFIGURE_GLOBALDEBUG_LEVEL_VERBOSE 0xfffffffu
enum FfxApiMsgType
{
FFX_API_MESSAGE_TYPE_ERROR = 0,
FFX_API_MESSAGE_TYPE_WARNING = 1,
FFX_API_MESSAGE_TYPE_COUNT
};
typedef void (*ffxApiMessage)(uint32_t type, const wchar_t* message);
#define FFX_API_CONFIGURE_DESC_TYPE_GLOBALDEBUG1 0x0000001u
struct ffxConfigureDescGlobalDebug1
{
ffxConfigureDescHeader header;
ffxApiMessage fpMessage;
uint32_t debugLevel;
};
#define FFX_API_CONFIGURE_DESC_TYPE_GLOBALDEBUG 7u
struct ffxConfigureDescGlobalDebug
{
ffxConfigureDescHeader header;
uint64_t effectId;
ffxApiMessage fpMessage; ///< A pointer to a function that can receive messages from the runtime. May be null.
uint32_t debugLevel;
};
#define FFX_API_QUERY_DESC_TYPE_GET_VERSIONS 4u
struct ffxQueryDescGetVersions
{
ffxQueryDescHeader header;
uint64_t createDescType; ///< Create description for the effect whose versions should be enumerated.
void* device; ///< For DX12: pointer to ID3D12Device.
uint64_t *outputCount; ///< Input capacity of id and name arrays. Output number of returned versions. If initially zero, output is number of available versions.
uint64_t *versionIds; ///< Output array of version ids to be used as version overrides. If null, only names and count are returned.
const char** versionNames; ///< Output array of version names for display. If null, only ids and count are returned. If both this and versionIds are null, only count is returned.
};
#define FFX_API_DESC_TYPE_OVERRIDE_VERSION 5u
struct ffxOverrideVersion
{
ffxApiHeader header;
uint64_t versionId; ///< Id of version to use. Must be a value returned from a query in ffxQueryDescGetVersions.versionIds array.
};
#define FFX_API_QUERY_DESC_TYPE_GET_PROVIDER_VERSION 6u
struct ffxQueryGetProviderVersion
{
ffxQueryDescHeader header;
uint64_t versionId; ///< Id of provider being used for queried context. 0 if invalid.
const char* versionName; ///< Version name for display. If nullptr, the query was invalid.
};
// Memory allocation function. Must return a valid pointer to at least size bytes of memory aligned to hold any type.
// May return null to indicate failure. Standard library malloc fulfills this requirement.
typedef void* (*ffxAlloc)(void* pUserData, uint64_t size);
// Memory deallocation function. May be called with null pointer as second argument.
typedef void (*ffxDealloc)(void* pUserData, void* pMem);
typedef struct ffxAllocationCallbacks
{
void* pUserData;
ffxAlloc alloc;
ffxDealloc dealloc;
} ffxAllocationCallbacks;
// Creates a FFX object context.
// Depending on the desc structures provided to this function, the context will be created with the desired version and attributes.
// Non-zero return indicates error code.
// Pointers passed in desc must remain live until ffxDestroyContext is called on the context.
// MemCb may be null; the system allocator (malloc/free) will be used in this case.
FFX_API_ENTRY ffxReturnCode_t ffxCreateContext(ffxContext* context, ffxCreateContextDescHeader* desc, const ffxAllocationCallbacks* memCb);
typedef ffxReturnCode_t (*PfnFfxCreateContext)(ffxContext* context, ffxCreateContextDescHeader* desc, const ffxAllocationCallbacks* memCb);
// Destroys an FFX object context.
// Non-zero return indicates error code.
// MemCb must be compatible with the callbacks passed into ffxCreateContext.
FFX_API_ENTRY ffxReturnCode_t ffxDestroyContext(ffxContext* context, const ffxAllocationCallbacks* memCb);
typedef ffxReturnCode_t (*PfnFfxDestroyContext)(ffxContext* context, const ffxAllocationCallbacks* memCb);
// Configures the provided FFX object context.
// If context is null, configure operates on any global state.
// Non-zero return indicates error code.
FFX_API_ENTRY ffxReturnCode_t ffxConfigure(ffxContext* context, const ffxConfigureDescHeader* desc);
typedef ffxReturnCode_t (*PfnFfxConfigure)(ffxContext* context, const ffxConfigureDescHeader* desc);
// Queries the provided FFX object context.
// If context is null, query operates on any global state.
// Non-zero return indicates error code.
FFX_API_ENTRY ffxReturnCode_t ffxQuery(ffxContext* context, ffxQueryDescHeader* desc);
typedef ffxReturnCode_t (*PfnFfxQuery)(ffxContext* context, ffxQueryDescHeader* desc);
// Dispatches work on the given FFX object context defined by the dispatch descriptor.
// Non-zero return indicates error code.
FFX_API_ENTRY ffxReturnCode_t ffxDispatch(ffxContext* context, const ffxDispatchDescHeader* desc);
typedef ffxReturnCode_t (*PfnFfxDispatch)(ffxContext* context, const ffxDispatchDescHeader* desc);
// FFX_API_EFFECT_IDs
#define FFX_API_EFFECT_ID_UPSCALE 0x00010000u
#define FFX_API_EFFECT_ID_FRAMEGENERATION 0x00020000u
#define FFX_API_EFFECT_ID_FRAMEGENERATIONSWAPCHAIN 0x00030000u
// Need to keep this ID around for the deprecated VK frame gen swapchain
#define FFX_API_EFFECT_ID_FRAMEGENERATIONSWAPCHAIN_VK 0x00040000u
// Need to keep this ID around for the deprecated VK frame gen swapchain
#define FFX_API_EFFECT_ID_DENOISER 0x00050000u
#define FFX_API_EFFECT_ID_RADIANCECACHE 0x00060000u
#define FFX_API_MAKE_EFFECT_SUB_ID(effectId, subversion) ((effectId & FFX_API_EFFECT_MASK) | (subversion & ~FFX_API_EFFECT_MASK))
// FFX_APID_BACKEND_IDs
#define FFX_API_BACKEND_ID_DX12 0x00000000u
#define FFX_API_BACKEND_ID_XBOX 0x01000000u
#define FFX_API_BACKEND_ID_VK 0x02000000u // For new effects going forward, please use this backend ID for vulkan specifics
#define FFX_API_MAKE_BACKEND_SUB_ID(backendId, subversion) ((backendId & FFX_API_BACKEND_MASK) | (subversion & ~FFX_API_BACKEND_MASK))
// Combiner for BACKEND-specific EFFECT sub-Ids
#define FFX_API_MAKE_BACKEND_EFFECT_SUB_ID(backendId, effectId, subversion) ((subversion & ~FFX_API_EFFECT_MASK) | (effectId & FFX_API_EFFECT_MASK) | (backendId & FFX_API_BACKEND_MASK) | (subversion & ~(FFX_API_BACKEND_MASK | FFX_API_EFFECT_MASK)))
// Pragma macros for controlling warnings so that deprecations take affect externally but can be suppressed internally.
// This is so we can maintain the API/ABI until we are ready to make the breaking change.
// These are sadly compiler specific.
#if defined(_MSC_VER) && !defined(__clang__) && !defined(__GNUC__) && !defined(__INTEL_COMPILER)
#define FFX_PRAGMA_WARNING_PUSH warning( push )
#define FFX_PRAGMA_WARNING_POP warning( pop )
#define FFX_PRAGMA_WARNING_DISABLE_DEPRECATIONS warning( disable: 4996 )
#define FFX_PRAGMA_WARNING_WARN_DEPRECATIONS warning( default: 4996 )
#elif defined(__clang__) || defined(__GNUC__)
#define FFX_PRAGMA_WARNING_PUSH GCC diagnostic push
#define FFX_PRAGMA_WARNING_POP GCC diagnostic pop
#define FFX_PRAGMA_WARNING_DISABLE_DEPRECATIONS GCC diagnostic ignored "-Wdeprecated"
#define FFX_PRAGMA_WARNING_WARN_DEPRECATIONS GCC diagnostic warning "-Wdeprecated"
#else
#define FFX_PRAGMA_WARNING_PUSH
#define FFX_PRAGMA_WARNING_POP
#define FFX_PRAGMA_WARNING_DISABLE_DEPRECATIONS
#endif
#define FFX_DEPRECATION(message) [[deprecated(message)]]
#if defined(__cplusplus)
}
#endif // #if defined(__cplusplus)
third_party/fidelityfx/api/include/ffx_api_types.h
Vendored
+257
View File
@@ -0,0 +1,257 @@
// This file is part of the FidelityFX SDK.
//
// Copyright (C) 2026 Advanced Micro Devices, Inc.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files(the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and /or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions :
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
#pragma once
#include <stdint.h>
#include <stdbool.h>
/// An enumeration of surface formats. Needs to match enum FfxApiSurfaceFormat
enum FfxApiSurfaceFormat
{
FFX_API_SURFACE_FORMAT_UNKNOWN, ///< Unknown format
FFX_API_SURFACE_FORMAT_R32G32B32A32_TYPELESS, ///< 32 bit per channel, 4 channel typeless format
FFX_API_SURFACE_FORMAT_R32G32B32A32_UINT, ///< 32 bit per channel, 4 channel uint format
FFX_API_SURFACE_FORMAT_R32G32B32A32_FLOAT, ///< 32 bit per channel, 4 channel float format
FFX_API_SURFACE_FORMAT_R16G16B16A16_FLOAT, ///< 16 bit per channel, 4 channel float format
FFX_API_SURFACE_FORMAT_R32G32B32_FLOAT, ///< 32 bit per channel, 3 channel float format
FFX_API_SURFACE_FORMAT_R32G32_FLOAT, ///< 32 bit per channel, 2 channel float format
FFX_API_SURFACE_FORMAT_R8_UINT, ///< 8 bit per channel, 1 channel float format
FFX_API_SURFACE_FORMAT_R32_UINT, ///< 32 bit per channel, 1 channel float format
FFX_API_SURFACE_FORMAT_R8G8B8A8_TYPELESS, ///< 8 bit per channel, 4 channel typeless format
FFX_API_SURFACE_FORMAT_R8G8B8A8_UNORM, ///< 8 bit per channel, 4 channel unsigned normalized format
FFX_API_SURFACE_FORMAT_R8G8B8A8_SNORM, ///< 8 bit per channel, 4 channel signed normalized format
FFX_API_SURFACE_FORMAT_R8G8B8A8_SRGB, ///< 8 bit per channel, 4 channel srgb normalized
FFX_API_SURFACE_FORMAT_B8G8R8A8_TYPELESS, ///< 8 bit per channel, 4 channel typeless format
FFX_API_SURFACE_FORMAT_B8G8R8A8_UNORM, ///< 8 bit per channel, 4 channel unsigned normalized format
FFX_API_SURFACE_FORMAT_B8G8R8A8_SRGB, ///< 8 bit per channel, 4 channel srgb normalized
FFX_API_SURFACE_FORMAT_R11G11B10_FLOAT, ///< 32 bit 3 channel float format
FFX_API_SURFACE_FORMAT_R10G10B10A2_UNORM, ///< 10 bit per 3 channel, 2 bit for 1 channel normalized format
FFX_API_SURFACE_FORMAT_R16G16_FLOAT, ///< 16 bit per channel, 2 channel float format
FFX_API_SURFACE_FORMAT_R16G16_UINT, ///< 16 bit per channel, 2 channel unsigned int format
FFX_API_SURFACE_FORMAT_R16G16_SINT, ///< 16 bit per channel, 2 channel signed int format
FFX_API_SURFACE_FORMAT_R16_FLOAT, ///< 16 bit per channel, 1 channel float format
FFX_API_SURFACE_FORMAT_R16_UINT, ///< 16 bit per channel, 1 channel unsigned int format
FFX_API_SURFACE_FORMAT_R16_UNORM, ///< 16 bit per channel, 1 channel unsigned normalized format
FFX_API_SURFACE_FORMAT_R16_SNORM, ///< 16 bit per channel, 1 channel signed normalized format
FFX_API_SURFACE_FORMAT_R8_UNORM, ///< 8 bit per channel, 1 channel unsigned normalized format
FFX_API_SURFACE_FORMAT_R8G8_UNORM, ///< 8 bit per channel, 2 channel unsigned normalized format
FFX_API_SURFACE_FORMAT_R8G8_UINT, ///< 8 bit per channel, 2 channel unsigned integer format
FFX_API_SURFACE_FORMAT_R32_FLOAT, ///< 32 bit per channel, 1 channel float format
FFX_API_SURFACE_FORMAT_R9G9B9E5_SHAREDEXP, ///< 9 bit per channel, 5 bit exponent format
FFX_API_SURFACE_FORMAT_R16G16B16A16_TYPELESS, ///< 16 bit per channel, 4 channel typeless format
FFX_API_SURFACE_FORMAT_R32G32_TYPELESS, ///< 32 bit per channel, 2 channel typeless format
FFX_API_SURFACE_FORMAT_R10G10B10A2_TYPELESS, ///< 10 bit per 3 channel, 2 bit for 1 channel typeless format
FFX_API_SURFACE_FORMAT_R16G16_TYPELESS, ///< 16 bit per channel, 2 channel typless format
FFX_API_SURFACE_FORMAT_R16_TYPELESS, ///< 16 bit per channel, 1 channel typeless format
FFX_API_SURFACE_FORMAT_R8_TYPELESS, ///< 8 bit per channel, 1 channel typeless format
FFX_API_SURFACE_FORMAT_R8G8_TYPELESS, ///< 8 bit per channel, 2 channel typeless format
FFX_API_SURFACE_FORMAT_R32_TYPELESS, ///< 32 bit per channel, 1 channel typeless format
FFX_API_SURFACE_FORMAT_R32G32_UINT, ///< 32 bit per channel, 2 channel uint format
FFX_API_SURFACE_FORMAT_R8_SNORM, ///< 8 bit per channel, 1 channel signed normalized format
};
/// An enumeration of resource usage.
enum FfxApiResourceUsage
{
FFX_API_RESOURCE_USAGE_READ_ONLY = 0, ///< No usage flags indicate a resource is read only.
FFX_API_RESOURCE_USAGE_RENDERTARGET = (1<<0), ///< Indicates a resource will be used as render target.
FFX_API_RESOURCE_USAGE_UAV = (1<<1), ///< Indicates a resource will be used as UAV.
FFX_API_RESOURCE_USAGE_DEPTHTARGET = (1<<2), ///< Indicates a resource will be used as depth target.
FFX_API_RESOURCE_USAGE_INDIRECT = (1<<3), ///< Indicates a resource will be used as indirect argument buffer
FFX_API_RESOURCE_USAGE_ARRAYVIEW = (1<<4), ///< Indicates a resource that will generate array views. Works on 2D and cubemap textures
FFX_API_RESOURCE_USAGE_STENCILTARGET = (1<<5), ///< Indicates a resource will be used as stencil target.
FFX_API_RESOURCE_USAGE_DCC_RENDERTARGET = (1<<15), ///< Indicates a resource that should specify optimal render target memory access flags (for console use)
};
/// An enumeration of resource states.
enum FfxApiResourceState
{
FFX_API_RESOURCE_STATE_COMMON = (1 << 0),
FFX_API_RESOURCE_STATE_UNORDERED_ACCESS = (1 << 1), ///< Indicates a resource is in the state to be used as UAV.
FFX_API_RESOURCE_STATE_COMPUTE_READ = (1 << 2), ///< Indicates a resource is in the state to be read by compute shaders.
FFX_API_RESOURCE_STATE_PIXEL_READ = (1 << 3), ///< Indicates a resource is in the state to be read by pixel shaders.
FFX_API_RESOURCE_STATE_PIXEL_COMPUTE_READ = (FFX_API_RESOURCE_STATE_PIXEL_READ | FFX_API_RESOURCE_STATE_COMPUTE_READ), ///< Indicates a resource is in the state to be read by pixel or compute shaders.
FFX_API_RESOURCE_STATE_COPY_SRC = (1 << 4), ///< Indicates a resource is in the state to be used as source in a copy command.
FFX_API_RESOURCE_STATE_COPY_DEST = (1 << 5), ///< Indicates a resource is in the state to be used as destination in a copy command.
FFX_API_RESOURCE_STATE_GENERIC_READ = (FFX_API_RESOURCE_STATE_COPY_SRC | FFX_API_RESOURCE_STATE_COMPUTE_READ), ///< Indicates a resource is in generic (slow) read state.
FFX_API_RESOURCE_STATE_INDIRECT_ARGUMENT = (1 << 6), ///< Indicates a resource is in the state to be used as an indirect command argument
FFX_API_RESOURCE_STATE_PRESENT = (1 << 7), ///< Indicates a resource is in the state to be used to present to the swap chain
FFX_API_RESOURCE_STATE_RENDER_TARGET = (1 << 8), ///< Indicates a resource is in the state to be used as render target
FFX_API_RESOURCE_STATE_DEPTH_ATTACHMENT = (1 << 9), ///< Indicates a resource is in the state to be used as depth attachment
};
/// An enumeration of surface dimensions.
enum FfxApiResourceDimension
{
FFX_API_RESOURCE_DIMENSION_TEXTURE_1D, ///< A resource with a single dimension.
FFX_API_RESOURCE_DIMENSION_TEXTURE_2D, ///< A resource with two dimensions.
};
/// An enumeration of resource flags.
enum FfxApiResourceFlags
{
FFX_API_RESOURCE_FLAGS_NONE = 0, ///< No flags.
FFX_API_RESOURCE_FLAGS_ALIASABLE = (1 << 0), ///< A bit indicating a resource does not need to persist across frames.
FFX_API_RESOURCE_FLAGS_UNDEFINED = (1 << 1), ///< Special case flag used internally when importing resources that require additional setup
};
// An enumeration for different resource types
enum FfxApiResourceType
{
FFX_API_RESOURCE_TYPE_BUFFER, ///< The resource is a buffer.
FFX_API_RESOURCE_TYPE_TEXTURE1D, ///< The resource is a 1-dimensional texture.
FFX_API_RESOURCE_TYPE_TEXTURE2D, ///< The resource is a 2-dimensional texture.
FFX_API_RESOURCE_TYPE_TEXTURE_CUBE, ///< The resource is a cube map.
FFX_API_RESOURCE_TYPE_TEXTURE3D, ///< The resource is a 3-dimensional texture.
};
enum FfxApiBackbufferTransferFunction
{
FFX_API_BACKBUFFER_TRANSFER_FUNCTION_SRGB,
FFX_API_BACKBUFFER_TRANSFER_FUNCTION_PQ,
FFX_API_BACKBUFFER_TRANSFER_FUNCTION_SCRGB
};
/// A structure encapsulating a 2-dimensional point, using 32bit unsigned integers.
struct FfxApiDimensions2D
{
uint32_t width; ///< The width of a 2-dimensional range.
uint32_t height; ///< The height of a 2-dimensional range.
};
/// A structure encapsulating a 2-dimensional set of floating point coordinates.
struct FfxApiFloatCoords2D
{
float x; ///< The x coordinate of a 2-dimensional point.
float y; ///< The y coordinate of a 2-dimensional point.
};
/// A structure encapsulating a 3-dimensional set of floating point coordinates.
struct FfxApiFloatCoords3D
{
float x; ///< The x coordinate of a 3-dimensional point.
float y; ///< The y coordinate of a 3-dimensional point.
float z; ///< The z coordinate of a 3-dimensional point.
};
/// A structure encapsulating a 2-dimensional rect.
struct FfxApiRect2D
{
int32_t left;
int32_t top;
int32_t width;
int32_t height;
};
/// A structure describing a resource.
///
/// @ingroup SDKTypes
struct FfxApiResourceDescription
{
uint32_t type; ///< The type of the resource.
uint32_t format; ///< The surface format.
union {
uint32_t width; ///< The width of the texture resource.
uint32_t size; ///< The size of the buffer resource.
};
union {
uint32_t height; ///< The height of the texture resource.
uint32_t stride; ///< The stride of the buffer resource.
};
union {
uint32_t depth; ///< The depth of the texture resource.
uint32_t alignment; ///< The alignment of the buffer resource.
};
uint32_t mipCount; ///< Number of mips (or 0 for full mipchain).
uint32_t flags; ///< A set of resource flags.
uint32_t usage; ///< Resource usage flags.
};
#define FFX_RESOURCE_NAME_SIZE 64
struct FfxApiResource
{
void* resource;
struct FfxApiResourceDescription description;
uint32_t state;
};
//struct definition matches FfxApiEffectMemoryUsage
typedef struct FfxApiEffectMemoryUsage
{
uint64_t totalUsageInBytes;
uint64_t aliasableUsageInBytes;
} FfxApiEffectMemoryUsage;
/// A structure describing a constant buffer allocation.
///
/// @ingroup SDKTypes
typedef struct FfxApiConstantBufferAllocation
{
struct FfxApiResource resource; ///< The resource representing the constant buffer resource.
uint64_t handle; ///< The binding handle for the constant buffer
} FfxApiConstantBufferAllocation;
/// A function definition for a constant buffer allocation callback
/// Used to provide a constant buffer allocator to the calling backend
///
/// @param [in] data The constant buffer data.
/// @param [in] dataSize The size of the constant buffer data.
///
/// @ingroup SDKTypes
typedef FfxApiConstantBufferAllocation(*FfxApiConstantBufferAllocator)(void* data, const uint64_t dataSize);
/// A 4-component floating point vector
typedef struct FfxApiFloat4
{
float x;
float y;
float z;
float w;
} FfxApiFloat4;
/// A 4x4 matrix in row-major layout with row-vector convention.
/// This is the canonical format expected by FFX APIs.
///
/// Memory layout: rows[row_index][column_index]
/// Mathematical convention: v * M (row vector multiplied by matrix)
typedef struct FfxApiMatrix4x4
{
FfxApiFloat4 rows[4]; ///< Four rows of the matrix. Access as rows[row].x, rows[row].y etc.
} FfxApiMatrix4x4;
/// An inclusive floating-point interval [min, max].
/// Valid instances must satisfy min <= max.
typedef struct FfxApiFloatBounds
{
float min; ///< Lower bound of the interval.
float max; ///< Upper bound of the interval.
} FfxApiFloatBounds;
third_party/fidelityfx/denoisers/include/ffx_denoiser.h
Vendored
+584
View File
@@ -0,0 +1,584 @@
// This file is part of the FidelityFX SDK.
//
// Copyright (C) 2026 Advanced Micro Devices, Inc.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files(the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and /or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions :
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
#pragma once
//------------------------------------------------------------------------------
// FFX Includes
//------------------------------------------------------------------------------
// FFX_API_EFFECT_ID_DENOISER
// FFX_API_MAKE_EFFECT_SUB_ID
// ffxConfigureDescHeader
// ffxCreateContextDescHeader
// ffxDispatchDescHeader
// ffxQueryDescHeader
#include "../../api/include/ffx_api.h"
// FfxApiDimensions2D
// FfxApiFloatBounds
// FfxApiFloatCoords2D
// FfxApiFloatCoords3D
// FfxApiMatrix4x4
// FfxApiResource
#include "../../api/include/ffx_api_types.h"
//------------------------------------------------------------------------------
// External Includes
//------------------------------------------------------------------------------
// uint32_t
// uint64_t
#include <stdint.h>
//------------------------------------------------------------------------------
// FFX Denoiser Defines
//------------------------------------------------------------------------------
/// Denoiser major version number
#define FFX_DENOISER_VERSION_MAJOR 1
/// Denoiser minor version number
#define FFX_DENOISER_VERSION_MINOR 2
/// Denoiser pinor version number
#define FFX_DENOISER_VERSION_PATCH 0
#define FFX_DENOISER_MAKE_VERSION(major, minor, patch) (((major) << 22) | ((minor) << 12) | (patch))
/// Denoiser combined version number
#define FFX_DENOISER_VERSION FFX_DENOISER_MAKE_VERSION(FFX_DENOISER_VERSION_MAJOR, FFX_DENOISER_VERSION_MINOR, FFX_DENOISER_VERSION_PATCH)
//------------------------------------------------------------------------------
// FFX Denoiser Declarations
//------------------------------------------------------------------------------
#ifdef __cplusplus
extern "C"
{
#endif // __cplusplus
//------------------------------------------------------------------------------
// FFX Denoiser Descriptions: Create Context
//------------------------------------------------------------------------------
#define FFX_API_CREATE_CONTEXT_DESC_TYPE_DENOISER FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x01)
/// Denoiser description for context creation.
typedef struct ffxCreateContextDescDenoiser
{
/// Header description
/// * @c type: FFX_API_CREATE_CONTEXT_DESC_TYPE_DENOISER
ffxCreateContextDescHeader header;
/// The version of the API the application was built against.
/// Must be set to @c FFX_DENOISER_VERSION.
uint32_t version;
/// Maximum size that rendering will be performed at.
struct FfxApiDimensions2D maxRenderSize;
/// Signal flags corresponding to a combination of values from @c FfxApiDenoiserSignalFlags.
uint32_t signalFlags;
/// Signal flags corresponding to a combination of values from @c FfxApiDenoiserSignalFlags
/// for signals that use checkerboard reconstruction.
/// Must be a subset of @c signalFlags.
uint32_t checkerboardSignalFlags;
/// Create flags corresponding to zero or a combination of values from @c FfxApiCreateContextDenoiserFlags.
uint32_t flags;
} ffxCreateContextDescDenoiser;
//------------------------------------------------------------------------------
// FFX Denoiser Descriptions: Configure
//------------------------------------------------------------------------------
#define FFX_API_CONFIGURE_DESC_TYPE_DENOISER_KEYVALUE FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x21)
/// Denoiser description for configuring.
/// @pre Required to be passed to @c ffxConfigure with a non-nullptr denoiser @c FfxContext*.
typedef struct ffxConfigureDescDenoiserKeyValue
{
/// Header description
/// * @c type: FFX_API_CONFIGURE_DESC_TYPE_DENOISER_KEYVALUE
ffxConfigureDescHeader header;
/// Configuration key corresponding to an entry of @c FfxApiConfigureDenoiserKey.
uint64_t key;
/// The number of elements to configure.
uint64_t count;
/// Pointer to an array with the configuration values for the elements to configure.
const void* data;
} ffxConfigureDescDenoiserKeyValue;
//------------------------------------------------------------------------------
// FFX Denoiser Descriptions: Dispatch
//------------------------------------------------------------------------------
typedef struct FfxApiDenoiserSignal
{
/// Input signal to be denoised.
struct FfxApiResource input;
/// Resulting denoised signal.
struct FfxApiResource output;
/// @brief The X-coordinate of the first traced pixel at Y=0 in a checkerboard pattern.
///
/// In a checkerboard pattern, only half the pixels are traced per frame, alternating as follows:
///
/// @code
/// x=0 x=1 x=2 x=3
/// y=0: 0 1 0 1
/// y=1: 1 0 1 0
/// y=2: 0 1 0 1
/// @endcode
///
/// * @c 0: pixel (0, 0) was traced.
/// * @c 1: pixel (1, 0) was traced.
///
/// @code checkerboardOrigin = frameIndex & 1 @endcode
///
/// @pre Must be zero-initialized if this signal is not set in @c checkerboardSignalFlags.
/// @pre Only meaningful if this signal is set in @c checkerboardSignalFlags.
uint32_t checkerboardOrigin;
} FfxApiDenoiserSignal;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x41)
/// Denoiser description for dispatching.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Always required to be passed to @c ffxDispatch.
typedef struct ffxDispatchDescDenoiser
{
/// Header description
///
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER
ffxDispatchDescHeader header;
/// A pointer to the command list.
///
/// * DX12: pointer to ID3D12GraphicsCommandList.
void* commandList;
/// @brief The signed linear depth.
///
/// Channels:
/// * R: Signed linear depth values for the current frame. @code CurrentLinearDepth @endcode
struct FfxApiResource linearDepth;
/// @brief The motion vectors.
///
/// Channels:
/// * RG: 2D motion vectors. @code PreviousUV - CurrentUV @endcode
/// * B: Signed linear depth delta. @code PreviousLinearDepth - CurrentLinearDepth @endcode
struct FfxApiResource motionVectors;
/// @brief The normals, roughness and material type.
///
/// Channels:
/// * RG: Octahedrally encoded normals
/// * B: Linear Roughness
/// * A: Material Type - See docs for more info,
struct FfxApiResource normals;
/// @brief The specular albedo.
///
/// Channels:
/// * RGB: Specular albedo - sqrt encoding assumed unless @c FFX_DENOISER_DISPATCH_NON_GAMMA_ALBEDO is provided. @code sqrt(SpecularAlbedo) @endcode
/// @pre Can be zero-initialized if and only if @c FFX_DENOISER_SIGNAL_DIRECT_SPECULAR, @c FFX_DENOISER_SIGNAL_INDIRECT_SPECULAR, and @c FFX_DENOISER_SIGNAL_DOMINANT_LIGHT_VISIBILITY are not set in the context creation signal flags.
struct FfxApiResource specularAlbedo;
/// @brief The diffuse albedo.
///
/// Channels:
/// * RGB: Diffuse albedo (e.g., @code BaseColor * (1 - Metalness)) @endcode - sqrt encoding assumed unless @c FFX_DENOISER_DISPATCH_NON_GAMMA_ALBEDO is provided. @code sqrt(DiffuseAlbedo) @endcode
/// @pre Can be zero-initialized if and only if @c FFX_DENOISER_SIGNAL_DIRECT_DIFFUSE, @c FFX_DENOISER_SIGNAL_INDIRECT_DIFFUSE, and @c FFX_DENOISER_SIGNAL_DOMINANT_LIGHT_VISIBILITY are not set in the context creation signal flags.
struct FfxApiResource diffuseAlbedo;
/// @brief The motion vector scale factor.
///
/// Channels:
/// * RG: The scale factor for transforming the 2D motion vectors into UV space
/// * For 2D motion vectors computed as @code PreviousUV - CurrentUV @endcode, use @code { .x = +1.0f, .y = +1.0f } @endcode.
/// * For 2D motion vectors computed as @code PreviousNDC - CurrentNDC @endcode, use @code { .x = +0.5f, .y = -0.5f } @endcode.
/// * B: The scale factor for transforming the signed linear depth delta
/// * For linear depth deltas computed as @code PreviousLinearDepth - CurrentLinearDepth @endcode, use @code { .z = +1.0f } @endcode.
struct FfxApiFloatCoords3D motionVectorScale;
/// The subpixel jitter offset applied to the camera projection. (Expressed in screen pixels)
struct FfxApiFloatCoords2D jitterOffsets;
/// The position delta of the camera since last frame. @code PreviousPosition - CurrentPosition @endcode (Expressed in world space)
struct FfxApiFloatCoords3D cameraPositionDelta;
/// @brief The world-to-view transformation matrix for the current frame.
///
/// Matrix convention:
/// * Storage: row-major (rows stored contiguously in memory)
/// * Multiplication: row vectors @code v' = vM @endcode
///
/// Compatibility:
/// * Column-major with column vectors (e.g., OpenGL, Cauldron):
/// direct copy (memcpy) - no transpose needed
/// * Column-major with row vectors:
/// requires transpose
/// * Row-major with row vectors (e.g., DirectXMath):
/// direct copy (memcpy)
/// * Row-major with column vectors:
/// requires transpose
///
/// Memory Layout Example (translation by (Tx, Ty, Tz)):
/// [m00 m01 m02 m03] [1 0 0 0]
/// [m10 m11 m12 m13] = [0 1 0 0]
/// [m20 m21 m22 m23] [0 0 1 0]
/// [m30 m31 m32 m33] [Tx Ty Tz 1]
FfxApiMatrix4x4 view;
/// @brief The (unjittered) view-to-projection transformation matrix for the current frame.
///
/// Matrix convention:
/// * Storage: row-major (rows stored contiguously in memory)
/// * Multiplication: row vectors @code v' = vM @endcode
///
/// Compatibility:
/// * Column-major with column vectors (e.g., OpenGL, Cauldron):
/// direct copy (memcpy) - no transpose needed
/// * Column-major with row vectors:
/// requires transpose
/// * Row-major with row vectors (e.g., DirectXMath):
/// direct copy (memcpy)
/// * Row-major with column vectors:
/// requires transpose
FfxApiMatrix4x4 projection;
/// @brief The absolute linear depth bounds considered for denoising.
/// @post Passthrough is enabled for signal texels where the corresponding absolute linear depth values are outside [linearDepthBounds.min,linearDepthBounds.max].
FfxApiFloatBounds linearDepthBounds;
/// The resolution that was used for rendering the input resources.
struct FfxApiDimensions2D renderSize;
/// The index of the current frame.
uint32_t frameIndex;
/// Dispatch flags corresponding to zero or a combination of values from @c FfxApiDispatchDenoiserFlags.
uint32_t flags;
} ffxDispatchDescDenoiser;
/// The maximum number of viewports that can be visualized.
#define FFX_API_DENOISER_DEBUG_VIEW_MAX_VIEWPORTS 12
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_DEBUG_VIEW FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x42)
/// Denoiser description for dispatching debug views.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Can be passed to @c ffxDispatch if and only if @c FFX_DENOISER_ENABLE_DEBUGGING is set in the context creation flags.
typedef struct ffxDispatchDescDenoiserDebugView
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_DEBUG_VIEW
ffxDispatchDescHeader header;
/// @brief The target output resource for debug visualization.
///
/// Channels:
/// * RGB: Color information to display.
/// * A: Indicates which pixels have been written to.
struct FfxApiResource output;
/// The resolution of the debug visualization.
struct FfxApiDimensions2D outputSize;
/// The debug view mode corresponding to an entry of @c FfxApiDenoiserDebugViewMode.
uint32_t mode;
/// The index of the viewport to visualize when @c FFX_API_DENOISER_DEBUG_VIEW_MODE_FULLSCREEN_VIEWPORT mode is active.
/// Clamped between 0 and @c FFX_API_DENOISER_DEBUG_VIEW_MAX_VIEWPORTS - 1.
uint32_t viewportIndex;
} ffxDispatchDescDenoiserDebugView;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_AMBIENT_OCCLUSION FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x43)
/// Denoiser description for dispatching ambient occlusion signals.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Required to be passed to @c ffxDispatch if and only if @c FFX_DENOISER_SIGNAL_AMBIENT_OCCLUSION is set in the context creation signal flags.
typedef struct ffxDispatchDescDenoiserAmbientOcclusion
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_AMBIENT_OCCLUSION
ffxDispatchDescHeader header;
/// Channels:
/// * R:
/// * @c input: Ambient occlusion (in [0,1] where 0 is fully occluded and 1 is fully exposed) to denoise
/// * @c output: Denoised ambient occlusion (in [0,1] where 0 is fully occluded and 1 is fully exposed) (or input if passthrough)
struct FfxApiDenoiserSignal signal;
} ffxDispatchDescDenoiserAmbientOcclusion;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_DIRECT_DIFFUSE FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x44)
/// Denoiser description for dispatching direct diffuse signals.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Required to be passed to @c ffxDispatch if and only if @c FFX_DENOISER_SIGNAL_DIRECT_DIFFUSE is set in the context creation signal flags.
typedef struct ffxDispatchDescDenoiserDirectDiffuse
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_DIRECT_DIFFUSE
ffxDispatchDescHeader header;
/// Channels:
/// * RGB:
/// * @c input: Direct diffuse radiance to denoise
/// * @c output: Denoised direct diffuse radiance (or input if passthrough)
/// * A:
/// * @c input: Undefined
/// * @c output: Preserved (or input if passthrough)
struct FfxApiDenoiserSignal signal;
} ffxDispatchDescDenoiserDirectDiffuse;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_DIRECT_SPECULAR FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x45)
/// Denoiser description for dispatching direct specular signals.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Required to be passed to @c ffxDispatch if and only if @c FFX_DENOISER_SIGNAL_DIRECT_SPECULAR is set in the context creation signal flags.
typedef struct ffxDispatchDescDenoiserDirectSpecular
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_DIRECT_SPECULAR
ffxDispatchDescHeader header;
/// Channels:
/// * RGB:
/// * @c input: Direct specular radiance to denoise
/// * @c output: Denoised direct specular radiance (or input if passthrough)
/// * A:
/// * @c input: Undefined
/// * @c output: Preserved (or input if passthrough)
struct FfxApiDenoiserSignal signal;
} ffxDispatchDescDenoiserDirectSpecular;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_DOMINANT_LIGHT FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x46)
/// Denoiser description for dispatching dominant light visibility signals.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Required to be passed to @c ffxDispatch if and only if @c FFX_DENOISER_SIGNAL_DOMINANT_LIGHT_VISIBILITY is set in the context creation signal flags.
typedef struct ffxDispatchDescDenoiserDominantLight
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_DOMINANT_LIGHT
ffxDispatchDescHeader header;
/// Channels:
/// * R:
/// * @c input: Dominant light visibility (ray hit distance [0,FP16_MAX] where FP16_MAX is fully exposed) to denoise
/// * @c output: Denoised dominant light visibility (in [0,1] where 0 is fully occluded and 1 is fully exposed) (or input if passthrough)
struct FfxApiDenoiserSignal signal;
/// Dominant light direction. (from light source to target)
struct FfxApiFloatCoords3D direction;
/// Dominant light emission. (i.e. @c LightColor * LightIntensity)
struct FfxApiFloatCoords3D emission;
/// Dominant light angular radius. (Expressed in radians)
float angularRadius;
} ffxDispatchDescDenoiserDominantLight;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_INDIRECT_DIFFUSE FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x47)
/// Denoiser description for dispatching indirect diffuse signals.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Required to be passed to @c ffxDispatch if and only if @c FFX_DENOISER_SIGNAL_INDIRECT_DIFFUSE is set in the context creation signal flags.
typedef struct ffxDispatchDescDenoiserIndirectDiffuse
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_INDIRECT_DIFFUSE
ffxDispatchDescHeader header;
/// Channels:
/// * RGB:
/// * @c input: Indirect diffuse radiance to denoise
/// * @c output: Denoised indirect diffuse radiance (or input if passthrough)
/// * A:
/// * @c input: Indirect diffuse ray hit distance
/// * @c output: Preserved (or input if passthrough)
struct FfxApiDenoiserSignal signal;
} ffxDispatchDescDenoiserIndirectDiffuse;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_INDIRECT_SPECULAR FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x48)
/// Denoiser description for dispatching of indirect specular signals.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Required to be passed to @c ffxDispatch if and only if @c FFX_DENOISER_SIGNAL_INDIRECT_SPECULAR is set in the context creation signal flags.
typedef struct ffxDispatchDescDenoiserIndirectSpecular
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_INDIRECT_SPECULAR
ffxDispatchDescHeader header;
/// Channels:
/// * RGB:
/// * @c input: Indirect specular radiance to denoise
/// * @c output: Denoised indirect specular radiance (or input if passthrough)
/// * A:
/// * @c input: Indirect specular ray hit distance
/// * @c output: Preserved (or input if passthrough)
struct FfxApiDenoiserSignal signal;
} ffxDispatchDescDenoiserIndirectSpecular;
#define FFX_API_DISPATCH_DESC_TYPE_DENOISER_SPECULAR_OCCLUSION FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x49)
/// Denoiser description for dispatching specular occlusion signals.
/// @pre Required to be passed to @c ffxDispatch with a non-nullptr denoiser @c FfxContext*.
/// @pre Required to be passed to @c ffxDispatch if and only if @c FFX_DENOISER_ENABLE_SPECULAR_OCCLUSION is set in the context creation signal flags.
typedef struct ffxDispatchDescDenoiserSpecularOcclusion
{
/// Header description
/// * @c type: FFX_API_DISPATCH_DESC_TYPE_DENOISER_SPECULAR_OCCLUSION
ffxDispatchDescHeader header;
/// Channels:
/// * R:
/// * @c input: Specular occlusion (in [0,1], where 0 is fully occluded and 1 is fully exposed) to denoise
/// * @c output: Denoised specular occlusion (in [0,1], where 0 is fully occluded and 1 is fully exposed) (or input if passthrough)
struct FfxApiDenoiserSignal signal;
} ffxDispatchDescDenoiserSpecularOcclusion;
//------------------------------------------------------------------------------
// FFX Denoiser Descriptions: Query
//------------------------------------------------------------------------------
#define FFX_API_QUERY_DESC_TYPE_DENOISER_GET_DEFAULT_KEYVALUE FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x81)
/// Denoiser description for querying default configuration values.
/// @pre Required to be passed to @c ffxQuery with a non-nullptr denoiser @c FfxContext*.
typedef struct ffxQueryDescDenoiserGetDefaultKeyValue
{
/// Header description
/// * @c type: FFX_API_QUERY_DESC_TYPE_DENOISER_GET_DEFAULT_KEYVALUE
ffxQueryDescHeader header;
/// Configuration key corresponding to an entry of @c FfxApiConfigureDenoiserKey.
uint64_t key;
/// The number of elements to query.
uint64_t count;
/// Pointer to an array for storing the queried elements.
void* data;
} ffxQueryDescDenoiserGetDefaultKeyValue;
#define FFX_API_QUERY_DESC_TYPE_DENOISER_GPU_MEMORY_USAGE FFX_API_MAKE_EFFECT_SUB_ID(FFX_API_EFFECT_ID_DENOISER, 0x82)
/// Denoiser description for querying GPU memory usage.
/// @pre Can be passed to @c ffxQuery with a non-nullptr denoiser @c FfxContext* to retrieve the actual memory usage.
/// @pre Can be passed to @c ffxQuery with a nullptr @c FfxContext* to retrieve the expected memory usage.
typedef struct ffxQueryDescDenoiserGetGPUMemoryUsage
{
/// Header description
/// * @c type: FFX_API_QUERY_DESC_TYPE_DENOISER_GPU_MEMORY_USAGE
ffxQueryDescHeader header;
/// A pointer to the device.
/// * DX12: pointer to ID3D12Device.
void* device;
/// Maximum size that rendering will be performed at.
struct FfxApiDimensions2D maxRenderSize;
/// Signal flags corresponding to a combination of values from @c FfxApiDenoiserSignalFlags.
uint32_t signalFlags;
/// Signal flags corresponding to a combination of values from @c FfxApiDenoiserSignalFlags
/// for signals that use checkerboard reconstruction.
/// Must be a subset of @c signalFlags.
uint32_t checkerboardSignalFlags;
/// Create flags corresponding to zero or a combination of values from @c FfxApiCreateContextDenoiserFlags.
uint32_t flags;
/// A pointer to a @c FfxApiEffectMemoryUsage structure that will hold the GPU memory usage.
struct FfxApiEffectMemoryUsage* gpuMemoryUsage;
} ffxQueryDescDenoiserGetGPUMemoryUsage;
//------------------------------------------------------------------------------
// FFX Denoiser Enums
//------------------------------------------------------------------------------
typedef enum FfxApiCreateContextDenoiserFlags
{
/// Flag indicating that debug features may be enabled. Memory usage may increase.
FFX_DENOISER_ENABLE_DEBUGGING = (1 << 0),
/// Flag indicating that exhaustive validation should be enabled. Performance may decrease.
FFX_DENOISER_ENABLE_VALIDATION = (1 << 1),
} FfxApiCreateContextDenoiserFlags;
typedef enum FfxApiConfigureDenoiserKey
{
/// Override the strength of the cross bilateral normal term.
/// * Format: A single @c float.
FFX_API_CONFIGURE_DENOISER_KEY_CROSS_BILATERAL_NORMAL_STRENGTH = 1,
/// Override the bias of the temporal accumulation to be more stable but less responsive.
/// * Format: A single @c float.
FFX_API_CONFIGURE_DENOISER_KEY_STABILITY_BIAS = 2,
/// Override the maximum radiance value. A single float scalar.
/// * Format: A single @c float.
FFX_API_CONFIGURE_DENOISER_KEY_MAX_RADIANCE = 3,
/// Override the standard deviation K value used for radiance clipping.
/// * Format: A single @c float.
FFX_API_CONFIGURE_DENOISER_KEY_RADIANCE_CLIP_STD_K = 4,
/// Override the Gaussian kernel relaxation factor.
/// * Format: A single @c float.
FFX_API_CONFIGURE_DENOISER_KEY_GAUSSIAN_KERNEL_RELAXATION = 5,
/// Override the discocclusion threshold used for depth comparisons during temporal reprojection.
/// * Format: A single @c float.
FFX_API_CONFIGURE_DENOISER_KEY_DISOCCLUSION_THRESHOLD = 6,
/// Override the absolute linear depth bounds used for normalizing the absolute linear depth debug view.
/// * Format: A single @c FfxApiFloatBounds.
FFX_API_CONFIGURE_DENOISER_KEY_DEBUG_VIEW_LINEAR_DEPTH_BOUNDS = 7,
} FfxApiConfigureDenoiserKey;
typedef enum FfxApiDenoiserDebugViewMode
{
/// Overview debug view mode.
FFX_API_DENOISER_DEBUG_VIEW_MODE_OVERVIEW = 0,
/// Fullscreen viewport debug view mode.
FFX_API_DENOISER_DEBUG_VIEW_MODE_FULLSCREEN_VIEWPORT = 1,
} FfxApiDenoiserDebugViewMode;
typedef enum FfxApiDenoiserSignalFlags
{
FFX_DENOISER_SIGNAL_NONE = 0,
/// Flag indicating that ambient occlusion needs to be denoised.
FFX_DENOISER_SIGNAL_AMBIENT_OCCLUSION = (1 << 0),
/// Flag indicating that direct diffuse needs to be denoised.
FFX_DENOISER_SIGNAL_DIRECT_DIFFUSE = (1 << 1),
/// Flag indicating that direct specular needs to be denoised.
FFX_DENOISER_SIGNAL_DIRECT_SPECULAR = (1 << 2),
/// Flag indicating that dominant light visibility needs to be denoised.
FFX_DENOISER_SIGNAL_DOMINANT_LIGHT_VISIBILITY = (1 << 3),
/// Flag indicating that indirect diffuse needs to be denoised.
FFX_DENOISER_SIGNAL_INDIRECT_DIFFUSE = (1 << 4),
/// Flag indicating that indirect specular needs to be denoised.
FFX_DENOISER_SIGNAL_INDIRECT_SPECULAR = (1 << 5),
/// Flag indicating that specular occlusion needs to be denoised.
FFX_DENOISER_SIGNAL_SPECULAR_OCCLUSION = (1 << 6),
} FfxApiDenoiserSignalFlags;
typedef enum FfxApiDispatchDenoiserFlags
{
/// Flag indicating that the history accumulation needs to be reset.
FFX_DENOISER_DISPATCH_RESET = (1 << 0),
/// Flag indicating that the (diffuse and specular) albedo textures are not gamma encoded.
FFX_DENOISER_DISPATCH_NON_GAMMA_ALBEDO = (1 << 1),
} FfxApiDispatchDenoiserFlags;
#ifdef __cplusplus
}
#endif // __cplusplus
tools/d4r_fsr_rr.h
+279
View File
@@ -0,0 +1,279 @@
// AMD FSR Ray Regeneration (FidelityFX denoiser 1.2) runtime access, shared by the NGX shim's
// FSR Ray Reconstruction backend and tools/fsr_rr_probe.cpp.
//
// The denoiser is AMD's signed amd_fidelityfx_denoiser_dx12.dll, loaded from an explicit path
// (never from a search path, never patched) and driven through its five FFX API exports. It
// records its compute work into the caller's D3D12 command list; nothing here involves CUDA,
// ZLUDA or NVIDIA's denoiser. Struct layouts come from AMD's MIT-licensed headers in
// third_party/fidelityfx (SDK 2.3.0 / FSR Ray Regeneration 1.2.0).
//
// Each stage reports separately (see FsrRrStage): a missing DLL, a missing export, a provider
// query that returns no denoiser for this device, a failed context creation (where shader
// pipelines are built) and a failed dispatch are different failures and are never folded together.
#pragma once
#include <windows.h>
#include <d3d12.h>
#include <cstddef>
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <string>
#include <vector>
#include "../third_party/fidelityfx/api/include/ffx_api.h"
#include "../third_party/fidelityfx/api/include/ffx_api_types.h"
#include "../third_party/fidelityfx/denoisers/include/ffx_denoiser.h"
// ffx_api_dx12.h's backend descriptor, declared here because that header's inline helpers call
// ID3D12Resource::GetDesc() by value, which does not compile with WIDL_EXPLICIT_AGGREGATE_RETURNS.
#ifndef FFX_API_CREATE_CONTEXT_DESC_TYPE_BACKEND_DX12
#define FFX_API_CREATE_CONTEXT_DESC_TYPE_BACKEND_DX12 FFX_API_MAKE_BACKEND_SUB_ID(FFX_API_BACKEND_ID_DX12, 0x02)
struct ffxCreateBackendDX12Desc
{
ffxCreateContextDescHeader header;
ID3D12Device* device;
};
#endif
static_assert(sizeof(ffxCreateBackendDX12Desc) == 24, "ffx_api_dx12.h layout");
// The 1.2 descriptors as the signed runtime reads them (x64, natural alignment, same as MSVC).
static_assert(sizeof(FfxApiResource) == 48 && sizeof(FfxApiDenoiserSignal) == 104, "ffx_api_types.h layout");
static_assert(sizeof(ffxCreateContextDescDenoiser) == 40, "ffx_denoiser.h 1.2 create layout");
static_assert(sizeof(ffxDispatchDescDenoiser) == 448 && offsetof(ffxDispatchDescDenoiser, view) == 296 &&
offsetof(ffxDispatchDescDenoiser, renderSize) == 432,
"ffx_denoiser.h 1.2 dispatch layout");
enum class FsrRrStage
{
NotLoaded, // nothing tried yet
DllMissing, // no file at the configured path
DllLoadFailed, // LoadLibrary failed (dependencies, architecture)
ExportsMissing, // not an FFX API DLL
NoProvider, // the runtime returned no denoiser version for this device
Ready, // a provider was enumerated for this device; contexts may still fail
};
inline const char* fsr_rr_stage_name(FsrRrStage stage)
{
switch (stage)
{
case FsrRrStage::NotLoaded: return "not loaded";
case FsrRrStage::DllMissing: return "denoiser DLL missing";
case FsrRrStage::DllLoadFailed: return "denoiser DLL failed to load";
case FsrRrStage::ExportsMissing: return "denoiser DLL lacks the FFX API exports";
case FsrRrStage::NoProvider: return "no FSR Ray Regeneration provider for this device";
case FsrRrStage::Ready: return "provider available";
}
return "?";
}
inline const char* fsr_rr_return_name(ffxReturnCode_t code)
{
switch (code)
{
case FFX_API_RETURN_OK: return "OK";
case FFX_API_RETURN_ERROR: return "ERROR";
case FFX_API_RETURN_ERROR_UNKNOWN_DESCTYPE: return "ERROR_UNKNOWN_DESCTYPE";
case FFX_API_RETURN_ERROR_RUNTIME_ERROR: return "ERROR_RUNTIME_ERROR";
case FFX_API_RETURN_NO_PROVIDER: return "NO_PROVIDER";
case FFX_API_RETURN_ERROR_MEMORY: return "ERROR_MEMORY";
case FFX_API_RETURN_ERROR_PARAMETER: return "ERROR_PARAMETER";
case FFX_API_RETURN_PROVIDER_NO_SUPPORT_NEW_DESCTYPE: return "PROVIDER_NO_SUPPORT_NEW_DESCTYPE";
}
return "unknown";
}
// The loaded runtime and the provider the device was offered. One per process: the DLL keeps
// global state (its debug callback), so it is loaded once and never unloaded while contexts exist.
struct FsrRrRuntime
{
HMODULE module = nullptr;
std::wstring path; // as configured
std::wstring mappedPath; // as the loader mapped it
PfnFfxCreateContext createContext = nullptr;
PfnFfxDestroyContext destroyContext = nullptr;
PfnFfxConfigure configure = nullptr;
PfnFfxQuery query = nullptr;
PfnFfxDispatch dispatch = nullptr;
FsrRrStage stage = FsrRrStage::NotLoaded;
ffxReturnCode_t queryResult = FFX_API_RETURN_OK;
std::vector<uint64_t> versionIds;
std::vector<std::string> versionNames;
uint64_t providerId = 0; // the provider contexts are created with (the first enumerated)
std::string providerName;
// Loads `dll` with its own directory searched for dependencies. Does not query providers.
bool load(const std::wstring& dll)
{
path = dll;
if (GetFileAttributesW(dll.c_str()) == INVALID_FILE_ATTRIBUTES)
{
stage = FsrRrStage::DllMissing;
return false;
}
module = LoadLibraryExW(dll.c_str(), nullptr, LOAD_WITH_ALTERED_SEARCH_PATH);
if (module == nullptr)
{
stage = FsrRrStage::DllLoadFailed;
return false;
}
wchar_t mapped[MAX_PATH] = {};
GetModuleFileNameW(module, mapped, MAX_PATH);
mappedPath = mapped;
createContext = reinterpret_cast<PfnFfxCreateContext>(reinterpret_cast<void*>(GetProcAddress(module, "ffxCreateContext")));
destroyContext = reinterpret_cast<PfnFfxDestroyContext>(reinterpret_cast<void*>(GetProcAddress(module, "ffxDestroyContext")));
configure = reinterpret_cast<PfnFfxConfigure>(reinterpret_cast<void*>(GetProcAddress(module, "ffxConfigure")));
query = reinterpret_cast<PfnFfxQuery>(reinterpret_cast<void*>(GetProcAddress(module, "ffxQuery")));
dispatch = reinterpret_cast<PfnFfxDispatch>(reinterpret_cast<void*>(GetProcAddress(module, "ffxDispatch")));
if (!createContext || !destroyContext || !configure || !query || !dispatch)
{
stage = FsrRrStage::ExportsMissing;
return false;
}
return true;
}
// Routes the runtime's own error/warning messages to `callback` (may be called on any thread).
ffxReturnCode_t set_message_callback(ffxApiMessage callback, uint32_t level)
{
ffxConfigureDescGlobalDebug desc = {};
desc.header.type = FFX_API_CONFIGURE_DESC_TYPE_GLOBALDEBUG;
desc.effectId = FFX_API_EFFECT_ID_DENOISER;
desc.fpMessage = callback;
desc.debugLevel = level;
return configure(nullptr, &desc.header);
}
// The two-call version enumeration AMD's sample uses, against the real device. Zero
// versions is the "no provider for this device" outcome, distinct from a query error.
bool query_providers(ID3D12Device* device)
{
versionIds.clear();
versionNames.clear();
uint64_t count = 0;
ffxQueryDescGetVersions desc = {};
desc.header.type = FFX_API_QUERY_DESC_TYPE_GET_VERSIONS;
desc.createDescType = FFX_API_EFFECT_ID_DENOISER;
desc.device = device;
desc.outputCount = &count;
queryResult = query(nullptr, &desc.header);
if (queryResult != FFX_API_RETURN_OK || count == 0)
{
stage = FsrRrStage::NoProvider;
return false;
}
versionIds.resize(count);
std::vector<const char*> names(count, nullptr);
desc.versionIds = versionIds.data();
desc.versionNames = names.data();
queryResult = query(nullptr, &desc.header);
if (queryResult != FFX_API_RETURN_OK || count == 0)
{
stage = FsrRrStage::NoProvider;
return false;
}
versionIds.resize(count);
for (uint64_t index = 0; index < count; ++index)
versionNames.emplace_back(names[index] != nullptr ? names[index] : "");
providerId = versionIds[0];
providerName = versionNames[0];
stage = FsrRrStage::Ready;
return true;
}
};
// GPU memory the denoiser would allocate for this configuration (total, aliasable).
inline ffxReturnCode_t fsr_rr_query_memory(FsrRrRuntime& runtime, ID3D12Device* device, uint32_t width,
uint32_t height, uint32_t signals, uint32_t flags,
FfxApiEffectMemoryUsage& usage)
{
ffxOverrideVersion version = {};
version.header.type = FFX_API_DESC_TYPE_OVERRIDE_VERSION;
version.versionId = runtime.providerId;
ffxQueryDescDenoiserGetGPUMemoryUsage desc = {};
desc.header.type = FFX_API_QUERY_DESC_TYPE_DENOISER_GPU_MEMORY_USAGE;
desc.header.pNext = &version.header;
desc.device = device;
desc.maxRenderSize = {width, height};
desc.signalFlags = signals;
desc.flags = flags;
desc.gpuMemoryUsage = &usage;
return runtime.query(nullptr, &desc.header);
}
// Creates one denoiser context on the enumerated provider. The descriptor chain only has to live
// for the call: ffx_api.h requires pointers *inside* descriptors to outlive the context, and
// none of these carry any.
inline ffxReturnCode_t fsr_rr_create_context(FsrRrRuntime& runtime, ID3D12Device* device, uint32_t maxWidth,
uint32_t maxHeight, uint32_t signals, uint32_t flags,
ffxContext& context)
{
ffxCreateContextDescDenoiser desc = {};
desc.header.type = FFX_API_CREATE_CONTEXT_DESC_TYPE_DENOISER;
desc.version = FFX_DENOISER_VERSION;
desc.maxRenderSize = {maxWidth, maxHeight};
desc.signalFlags = signals;
desc.flags = flags;
ffxCreateBackendDX12Desc backend = {};
backend.header.type = FFX_API_CREATE_CONTEXT_DESC_TYPE_BACKEND_DX12;
backend.device = device;
ffxOverrideVersion version = {};
version.header.type = FFX_API_DESC_TYPE_OVERRIDE_VERSION;
version.versionId = runtime.providerId;
desc.header.pNext = &backend.header;
backend.header.pNext = &version.header;
context = nullptr;
return runtime.createContext(&context, &desc.header, nullptr);
}
// Asks a live context which provider actually serves it (the runtime's own account, logged as
// the "actual backend").
inline std::string fsr_rr_context_provider(FsrRrRuntime& runtime, ffxContext& context, uint64_t* id = nullptr)
{
ffxQueryGetProviderVersion desc = {};
desc.header.type = FFX_API_QUERY_DESC_TYPE_GET_PROVIDER_VERSION;
if (runtime.query(&context, &desc.header) != FFX_API_RETURN_OK || desc.versionName == nullptr)
return std::string();
if (id != nullptr)
*id = desc.versionId;
return desc.versionName;
}
// An FfxApiResource for a 2D texture whose description the caller already knows (the shim never
// calls GetDesc() by value; see above).
inline FfxApiResource fsr_rr_texture(ID3D12Resource* resource, const D3D12_RESOURCE_DESC& desc, uint32_t format,
uint32_t state)
{
FfxApiResource result = {};
result.resource = resource;
result.state = state;
if (resource == nullptr)
return result;
result.description.type = FFX_API_RESOURCE_TYPE_TEXTURE2D;
result.description.format = format;
result.description.width = static_cast<uint32_t>(desc.Width);
result.description.height = desc.Height;
result.description.depth = desc.DepthOrArraySize;
result.description.mipCount = desc.MipLevels;
result.description.usage = FFX_API_RESOURCE_USAGE_READ_ONLY;
if (desc.Flags & D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS)
result.description.usage |= FFX_API_RESOURCE_USAGE_UAV;
return result;
}
// The FFX surface formats the adapter's own intermediate textures use.
inline uint32_t fsr_rr_surface_format(DXGI_FORMAT format)
{
switch (format)
{
case DXGI_FORMAT_R32_FLOAT: return FFX_API_SURFACE_FORMAT_R32_FLOAT;
case DXGI_FORMAT_R16G16B16A16_FLOAT: return FFX_API_SURFACE_FORMAT_R16G16B16A16_FLOAT;
case DXGI_FORMAT_R32G32B32A32_FLOAT: return FFX_API_SURFACE_FORMAT_R32G32B32A32_FLOAT;
case DXGI_FORMAT_R10G10B10A2_UNORM: return FFX_API_SURFACE_FORMAT_R10G10B10A2_UNORM;
case DXGI_FORMAT_R8G8B8A8_UNORM: return FFX_API_SURFACE_FORMAT_R8G8B8A8_UNORM;
case DXGI_FORMAT_R16_FLOAT: return FFX_API_SURFACE_FORMAT_R16_FLOAT;
case DXGI_FORMAT_R8_UNORM: return FFX_API_SURFACE_FORMAT_R8_UNORM;
default: return FFX_API_SURFACE_FORMAT_UNKNOWN;
}
}
tools/d4r_fsr_rr_backend.inl
+1156
View File
File diff suppressed because it is too large. Load diff
tools/d4r_fsr_rr_dispatch_profile.h
+777
View File
@@ -0,0 +1,777 @@
// Diagnostic per-dispatch GPU timing of AMD's denoiser (D4R_FSR_RR_DISPATCH_PROFILE=1, off by default).
//
// While ffxDispatch records into the game's list, the list's Dispatch/ExecuteIndirect entries (vkd3d-proton's
// shared command-list vtable) are wrapped to put a timestamp before every dispatch; the device's compute PSO
// creation is wrapped to tag each pipeline with vkd3d's shader hash (FNV-1 64 of the DXBC blob, the name
// VKD3D_SHADER_DUMP_PATH and VKD3D_SHADER_OVERRIDE use). Stamp i..i+1 is the time from the end of the work
// before dispatch i to the end of dispatch i (bottom-of-pipe), which is dispatch i's cost when the denoiser
// separates its passes with barriers. Wrappers forward unchanged outside the recording thread's window.
#pragma once
#include <d3d12.h>
#include <algorithm>
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <string>
#include <map>
#include <mutex>
#include <unordered_map>
#include <vector>
namespace fsr_rr_dp
{
constexpr UINT kSlots = 4;
constexpr UINT kStampsPerSlot = 2048;
using DispatchFn = void(STDMETHODCALLTYPE*)(ID3D12GraphicsCommandList*, UINT, UINT, UINT);
using ExecuteIndirectFn = void(STDMETHODCALLTYPE*)(ID3D12GraphicsCommandList*, ID3D12CommandSignature*, UINT,
ID3D12Resource*, UINT64, ID3D12Resource*, UINT64);
using ResourceBarrierFn = void(STDMETHODCALLTYPE*)(ID3D12GraphicsCommandList*, UINT, const D3D12_RESOURCE_BARRIER*);
using SetRootCbvFn = void(STDMETHODCALLTYPE*)(ID3D12GraphicsCommandList*, UINT, D3D12_GPU_VIRTUAL_ADDRESS);
using SetPipelineStateFn = void(STDMETHODCALLTYPE*)(ID3D12GraphicsCommandList*, ID3D12PipelineState*);
using CreateComputeFn = HRESULT(STDMETHODCALLTYPE*)(ID3D12Device*, const D3D12_COMPUTE_PIPELINE_STATE_DESC*, REFIID,
void**);
using CreateStreamFn = HRESULT(STDMETHODCALLTYPE*)(ID3D12Device*, const D3D12_PIPELINE_STATE_STREAM_DESC*, REFIID,
void**);
struct Record
{
uint64_t hash;
UINT x, y, z;
bool indirect;
uint32_t barriers, uavBarriers; // recorded by the denoiser between the previous dispatch and this one
};
struct Stat
{
uint64_t hash = 0;
UINT x = 0, y = 0, z = 0;
bool indirect = false;
uint32_t barriers = 0, uavBarriers = 0;
double sum = 0;
uint32_t count = 0;
};
struct State
{
std::mutex mutex;
std::unordered_map<const void*, uint64_t> psoHash;
DispatchFn dispatch = nullptr;
ExecuteIndirectFn executeIndirect = nullptr;
SetPipelineStateFn setPipelineState = nullptr;
ResourceBarrierFn resourceBarrier = nullptr;
SetRootCbvFn setRootCbv = nullptr;
uint32_t barriersSinceDispatch = 0, uavBarriersSinceDispatch = 0;
CreateComputeFn createCompute = nullptr;
CreateStreamFn createStream = nullptr;
bool listHooked = false, deviceHooked = false, failed = false;
ID3D12QueryHeap* heap = nullptr;
ID3D12Resource* readback = nullptr;
volatile uint64_t* values = nullptr;
double ticksPerMs = 0;
std::vector<Record> records[kSlots];
UINT used[kSlots] = {};
std::map<UINT, Stat> stats; // by dispatch index within the frame
uint32_t frames = 0;
uint64_t lastHash = 0;
};
inline State& state()
{
static State s;
return s;
}
thread_local ID3D12GraphicsCommandList* t_list = nullptr;
thread_local UINT t_slot = 0;
thread_local uint64_t t_pso = 0;
thread_local D3D12_GPU_VIRTUAL_ADDRESS t_cbv = 0;
inline void snapshot_after_dispatch(ID3D12GraphicsCommandList* list);
inline uint64_t fnv1(const void* data, size_t size)
{
uint64_t h = 0xcbf29ce484222325ull;
const auto* p = static_cast<const uint8_t*>(data);
for (size_t i = 0; i < size; ++i)
h = (h * 0x100000001b3ull) ^ p[i];
return h;
}
inline bool patch(void** slot, void* hook, void** original)
{
if (*slot == hook)
return true;
DWORD old = 0;
if (!VirtualProtect(slot, sizeof(void*), PAGE_READWRITE, &old))
return false;
*original = *slot;
*slot = hook;
VirtualProtect(slot, sizeof(void*), old, &old);
return true;
}
inline void stamp(ID3D12GraphicsCommandList* list, uint64_t hash, UINT x, UINT y, UINT z, bool indirect)
{
State& s = state();
UINT& used = s.used[t_slot];
if (used + 1 >= kStampsPerSlot)
return;
list->EndQuery(s.heap, D3D12_QUERY_TYPE_TIMESTAMP, t_slot * kStampsPerSlot + used);
s.records[t_slot].push_back({hash, x, y, z, indirect, s.barriersSinceDispatch, s.uavBarriersSinceDispatch});
s.barriersSinceDispatch = s.uavBarriersSinceDispatch = 0;
++used;
}
static void STDMETHODCALLTYPE hook_set_pipeline_state(ID3D12GraphicsCommandList* list, ID3D12PipelineState* pso)
{
State& s = state();
if (t_list == list)
{
std::lock_guard<std::mutex> lock(s.mutex);
auto found = s.psoHash.find(pso);
t_pso = found != s.psoHash.end() ? found->second : 0;
}
s.setPipelineState(list, pso);
}
inline bool serialize()
{
static const bool on = env_uint("D4R_FSR_RR_DP_SERIALIZE", 0) != 0;
return on;
}
static void STDMETHODCALLTYPE hook_resource_barrier(ID3D12GraphicsCommandList* list, UINT count,
const D3D12_RESOURCE_BARRIER* barriers)
{
State& s = state();
if (t_list == list)
for (UINT i = 0; i < count; ++i)
{
++s.barriersSinceDispatch;
if (barriers[i].Type == D3D12_RESOURCE_BARRIER_TYPE_UAV)
++s.uavBarriersSinceDispatch;
}
s.resourceBarrier(list, count, barriers);
}
// Diagnostic: a full UAV barrier before every dispatch of the denoiser (tests for missing barriers).
inline void serialize_barrier(ID3D12GraphicsCommandList* list)
{
if (!serialize())
return;
D3D12_RESOURCE_BARRIER uav = {};
uav.Type = D3D12_RESOURCE_BARRIER_TYPE_UAV;
state().resourceBarrier(list, 1, &uav);
}
static void STDMETHODCALLTYPE hook_set_root_cbv(ID3D12GraphicsCommandList* list, UINT index,
D3D12_GPU_VIRTUAL_ADDRESS va)
{
if (t_list == list)
t_cbv = va;
state().setRootCbv(list, index, va);
}
static void STDMETHODCALLTYPE hook_dispatch(ID3D12GraphicsCommandList* list, UINT x, UINT y, UINT z)
{
State& s = state();
if (t_list == list)
{
serialize_barrier(list);
stamp(list, t_pso, x, y, z, false);
}
s.dispatch(list, x, y, z);
if (t_list == list)
snapshot_after_dispatch(list);
}
static void STDMETHODCALLTYPE hook_execute_indirect(ID3D12GraphicsCommandList* list, ID3D12CommandSignature* sig,
UINT maxCount, ID3D12Resource* args, UINT64 argsOffset,
ID3D12Resource* count, UINT64 countOffset)
{
State& s = state();
if (t_list == list)
{
serialize_barrier(list);
stamp(list, t_pso, maxCount, 0, 0, true);
}
s.executeIndirect(list, sig, maxCount, args, argsOffset, count, countOffset);
if (t_list == list)
snapshot_after_dispatch(list);
}
static HRESULT STDMETHODCALLTYPE hook_create_compute(ID3D12Device* device, const D3D12_COMPUTE_PIPELINE_STATE_DESC* desc,
REFIID riid, void** pso)
{
State& s = state();
const HRESULT hr = s.createCompute(device, desc, riid, pso);
if (SUCCEEDED(hr) && pso != nullptr && *pso != nullptr && desc != nullptr && desc->CS.pShaderBytecode != nullptr)
{
const uint64_t hash = fnv1(desc->CS.pShaderBytecode, desc->CS.BytecodeLength);
std::lock_guard<std::mutex> lock(s.mutex);
s.psoHash[*pso] = hash;
}
return hr;
}
// The CS bytecode of a pipeline state stream (D3DX12-style subobjects, each aligned to a pointer).
inline const D3D12_SHADER_BYTECODE* stream_cs(const D3D12_PIPELINE_STATE_STREAM_DESC* desc)
{
auto* p = static_cast<const uint8_t*>(desc->pPipelineStateSubobjectStream);
const uint8_t* end = p + desc->SizeInBytes;
while (p + sizeof(UINT) <= end)
{
const UINT type = *reinterpret_cast<const UINT*>(p);
size_t size;
switch (type)
{
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_ROOT_SIGNATURE: size = 16; break;
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_CS:
return reinterpret_cast<const D3D12_SHADER_BYTECODE*>(p + 8);
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_VS:
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_PS:
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_DS:
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_HS:
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_GS:
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_CACHED_PSO: size = 24; break;
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_NODE_MASK:
case D3D12_PIPELINE_STATE_SUBOBJECT_TYPE_FLAGS: size = 8; break;
default: return nullptr;
}
p += size;
}
return nullptr;
}
static HRESULT STDMETHODCALLTYPE hook_create_stream(ID3D12Device* device, const D3D12_PIPELINE_STATE_STREAM_DESC* desc,
REFIID riid, void** pso)
{
State& s = state();
const HRESULT hr = s.createStream(device, desc, riid, pso);
const D3D12_SHADER_BYTECODE* cs = desc != nullptr ? stream_cs(desc) : nullptr;
if (SUCCEEDED(hr) && pso != nullptr && *pso != nullptr && cs != nullptr && cs->pShaderBytecode != nullptr)
{
const uint64_t hash = fnv1(cs->pShaderBytecode, cs->BytecodeLength);
std::lock_guard<std::mutex> lock(s.mutex);
s.psoHash[*pso] = hash;
}
return hr;
}
// D4R_FSR_RR_DP_SNAPSHOT=k1,k2,...: on frame D4R_FSR_RR_DP_SNAPSHOT_FRAME (default 1), copy every UAV buffer the
// denoiser created after its dispatch k (0-based), to <D4R_FSR_RR_DUMP_DIR>/snap_<k>_<buffer>.raw at teardown.
using CreateCommittedFn = HRESULT(STDMETHODCALLTYPE*)(ID3D12Device*, const D3D12_HEAP_PROPERTIES*, D3D12_HEAP_FLAGS,
const D3D12_RESOURCE_DESC*, D3D12_RESOURCE_STATES,
const D3D12_CLEAR_VALUE*, REFIID, void**);
using CreatePlacedFn = HRESULT(STDMETHODCALLTYPE*)(ID3D12Device*, ID3D12Heap*, UINT64, const D3D12_RESOURCE_DESC*,
D3D12_RESOURCE_STATES, const D3D12_CLEAR_VALUE*, REFIID, void**);
using CreateCommitted1Fn = HRESULT(STDMETHODCALLTYPE*)(ID3D12Device*, const D3D12_HEAP_PROPERTIES*, D3D12_HEAP_FLAGS,
const D3D12_RESOURCE_DESC*, D3D12_RESOURCE_STATES,
const D3D12_CLEAR_VALUE*, ID3D12ProtectedResourceSession*,
REFIID, void**);
struct Created
{
ID3D12Resource* resource;
D3D12_RESOURCE_DESC desc;
const char* how;
};
inline std::vector<Created>& created()
{
static std::vector<Created> c;
return c;
}
thread_local bool t_creating = false;
inline CreateCommittedFn& orig_committed() { static CreateCommittedFn f = nullptr; return f; }
inline CreatePlacedFn& orig_placed() { static CreatePlacedFn f = nullptr; return f; }
inline CreateCommitted1Fn& orig_committed1() { static CreateCommitted1Fn f = nullptr; return f; }
inline void note_created(HRESULT hr, const D3D12_RESOURCE_DESC* desc, void** out, const char* how)
{
if (!t_creating || FAILED(hr) || out == nullptr || *out == nullptr || desc == nullptr)
return;
// Only ID3D12Resource-compatible pointers are recorded; every resource interface starts with ID3D12Resource.
created().push_back({static_cast<ID3D12Resource*>(*out), *desc, how});
logf("D4R_FSR_RR_DP resource %zu (%s): dim %d %llux%ux%u format %d flags 0x%x", created().size() - 1, how,
desc->Dimension, static_cast<unsigned long long>(desc->Width), desc->Height, desc->DepthOrArraySize,
desc->Format, desc->Flags);
}
static HRESULT STDMETHODCALLTYPE hook_committed(ID3D12Device* d, const D3D12_HEAP_PROPERTIES* hp, D3D12_HEAP_FLAGS hf,
const D3D12_RESOURCE_DESC* desc, D3D12_RESOURCE_STATES st,
const D3D12_CLEAR_VALUE* cv, REFIID riid, void** out)
{
const HRESULT hr = orig_committed()(d, hp, hf, desc, st, cv, riid, out);
note_created(hr, desc, out, "committed");
return hr;
}
static HRESULT STDMETHODCALLTYPE hook_placed(ID3D12Device* d, ID3D12Heap* heap, UINT64 offset,
const D3D12_RESOURCE_DESC* desc, D3D12_RESOURCE_STATES st,
const D3D12_CLEAR_VALUE* cv, REFIID riid, void** out)
{
const HRESULT hr = orig_placed()(d, heap, offset, desc, st, cv, riid, out);
note_created(hr, desc, out, "placed");
return hr;
}
static HRESULT STDMETHODCALLTYPE hook_committed1(ID3D12Device* d, const D3D12_HEAP_PROPERTIES* hp, D3D12_HEAP_FLAGS hf,
const D3D12_RESOURCE_DESC* desc, D3D12_RESOURCE_STATES st,
const D3D12_CLEAR_VALUE* cv, ID3D12ProtectedResourceSession* ps,
REFIID riid, void** out)
{
const HRESULT hr = orig_committed1()(d, hp, hf, desc, st, cv, ps, riid, out);
note_created(hr, desc, out, "committed1");
return hr;
}
struct Snapshot
{
UINT dispatch;
std::vector<std::pair<size_t, ID3D12Resource*>> copies; // created() index, readback buffer
};
inline std::vector<Snapshot>& snapshots()
{
static std::vector<Snapshot> s;
return s;
}
inline const std::vector<UINT>& snapshot_points()
{
static const std::vector<UINT> points = [] {
std::vector<UINT> p;
const std::string list = env_string("D4R_FSR_RR_DP_SNAPSHOT");
size_t at = 0;
while (at < list.size())
{
const size_t comma = list.find(',', at);
p.push_back(static_cast<UINT>(std::stoul(list.substr(at, comma - at))));
at = comma == std::string::npos ? list.size() : comma + 1;
}
return p;
}();
return points;
}
thread_local uint32_t t_frame = 0;
thread_local UINT t_dispatchIndex = 0;
thread_local ID3D12Device* t_device = nullptr;
inline bool enabled()
{
static const bool on = env_uint("D4R_FSR_RR_DISPATCH_PROFILE", 0) != 0;
return on;
}
// Before the denoiser's context is created: tag the PSOs it builds.
inline void hook_device(ID3D12Device* device)
{
State& s = state();
if (!enabled() || s.deviceHooked)
return;
void** vtable = *reinterpret_cast<void***>(device);
ID3D12Device2* device2 = nullptr;
bool ok = patch(&vtable[11], reinterpret_cast<void*>(&hook_create_compute), reinterpret_cast<void**>(&s.createCompute));
if (SUCCEEDED(device->QueryInterface(__uuidof(ID3D12Device2), reinterpret_cast<void**>(&device2))))
{
void** vtable2 = *reinterpret_cast<void***>(device2);
ok = ok && patch(&vtable2[47], reinterpret_cast<void*>(&hook_create_stream), reinterpret_cast<void**>(&s.createStream));
device2->Release();
}
if (!snapshot_points().empty())
{
ID3D12Device4* device4 = nullptr;
ok = ok && patch(&vtable[27], reinterpret_cast<void*>(&hook_committed), reinterpret_cast<void**>(&orig_committed())) &&
patch(&vtable[29], reinterpret_cast<void*>(&hook_placed), reinterpret_cast<void**>(&orig_placed()));
if (SUCCEEDED(device->QueryInterface(__uuidof(ID3D12Device4), reinterpret_cast<void**>(&device4))))
{
void** vtable4 = *reinterpret_cast<void***>(device4);
ok = ok && patch(&vtable4[53], reinterpret_cast<void*>(&hook_committed1),
reinterpret_cast<void**>(&orig_committed1()));
device4->Release();
}
}
s.deviceHooked = ok;
logf("D4R_FSR_RR_DISPATCH_PROFILE: PSO creation hooks %s", ok ? "installed" : "FAILED");
}
inline bool ensure(ID3D12Device* device, ID3D12GraphicsCommandList* list)
{
State& s = state();
if (s.failed)
return false;
if (!s.listHooked)
{
void** vtable = *reinterpret_cast<void***>(list);
const bool ok =
patch(&vtable[14], reinterpret_cast<void*>(&hook_dispatch), reinterpret_cast<void**>(&s.dispatch)) &&
patch(&vtable[25], reinterpret_cast<void*>(&hook_set_pipeline_state),
reinterpret_cast<void**>(&s.setPipelineState)) &&
patch(&vtable[59], reinterpret_cast<void*>(&hook_execute_indirect),
reinterpret_cast<void**>(&s.executeIndirect)) &&
patch(&vtable[26], reinterpret_cast<void*>(&hook_resource_barrier),
reinterpret_cast<void**>(&s.resourceBarrier)) &&
patch(&vtable[37], reinterpret_cast<void*>(&hook_set_root_cbv), reinterpret_cast<void**>(&s.setRootCbv));
if (!ok)
{
s.failed = true;
logf("D4R_FSR_RR_DISPATCH_PROFILE: command-list hooks FAILED");
return false;
}
s.listHooked = true;
}
if (s.heap == nullptr)
{
D3D12_COMMAND_QUEUE_DESC queueDesc = {};
queueDesc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT;
ID3D12CommandQueue* queue = nullptr;
UINT64 frequency = 0;
if (SUCCEEDED(device->CreateCommandQueue(&queueDesc, __uuidof(ID3D12CommandQueue), reinterpret_cast<void**>(&queue))))
{
queue->GetTimestampFrequency(&frequency);
queue->Release();
}
D3D12_QUERY_HEAP_DESC heapDesc = {};
heapDesc.Type = D3D12_QUERY_HEAP_TYPE_TIMESTAMP;
heapDesc.Count = kSlots * kStampsPerSlot;
D3D12_HEAP_PROPERTIES heap = {};
heap.Type = D3D12_HEAP_TYPE_READBACK;
D3D12_RESOURCE_DESC desc = {};
desc.Dimension = D3D12_RESOURCE_DIMENSION_BUFFER;
desc.Width = 8ull * kSlots * kStampsPerSlot;
desc.Height = 1;
desc.DepthOrArraySize = 1;
desc.MipLevels = 1;
desc.SampleDesc.Count = 1;
desc.Layout = D3D12_TEXTURE_LAYOUT_ROW_MAJOR;
void* mapped = nullptr;
if (frequency == 0 ||
FAILED(device->CreateQueryHeap(&heapDesc, __uuidof(ID3D12QueryHeap), reinterpret_cast<void**>(&s.heap))) ||
FAILED(device->CreateCommittedResource(&heap, D3D12_HEAP_FLAG_NONE, &desc, D3D12_RESOURCE_STATE_COPY_DEST,
nullptr, __uuidof(ID3D12Resource),
reinterpret_cast<void**>(&s.readback))) ||
FAILED(s.readback->Map(0, nullptr, &mapped)))
{
s.failed = true;
logf("D4R_FSR_RR_DISPATCH_PROFILE: query heap FAILED");
return false;
}
s.values = static_cast<volatile uint64_t*>(mapped);
for (UINT i = 0; i < kSlots * kStampsPerSlot; ++i)
s.values[i] = 0;
s.ticksPerMs = static_cast<double>(frequency) / 1000.0;
}
return true;
}
inline void log_stats()
{
State& s = state();
double total = 0;
for (const auto& [index, stat] : s.stats)
total += stat.sum / std::max(1u, stat.count);
logf("D4R_FSR_RR_DISPATCH_PROFILE: %u frames, %zu dispatches/frame, %.3f ms GPU/frame (mean)", s.frames,
s.stats.size(), total);
for (const auto& [index, stat] : s.stats)
logf("D4R_FSR_RR_DISPATCH %4u %016llx %s %5u %5u %5u %.4f ms (n=%u) barriers %u uav %u", index,
static_cast<unsigned long long>(stat.hash), stat.indirect ? "ind" : "dir", stat.x, stat.y, stat.z,
stat.sum / std::max(1u, stat.count), stat.count, stat.barriers, stat.uavBarriers);
std::map<uint64_t, std::pair<double, uint32_t>> byHash;
for (const auto& [index, stat] : s.stats)
{
auto& entry = byHash[stat.hash];
entry.first += stat.sum / std::max(1u, stat.count);
++entry.second;
}
std::vector<std::pair<double, uint64_t>> ranked;
for (const auto& [hash, entry] : byHash)
ranked.push_back({entry.first, hash});
std::sort(ranked.rbegin(), ranked.rend());
for (const auto& [ms, hash] : ranked)
logf("D4R_FSR_RR_SHADER %016llx %.4f ms/frame over %u dispatches (%.1f%%)", static_cast<unsigned long long>(hash),
ms, byHash[hash].second, total > 0 ? 100.0 * ms / total : 0.0);
}
// Around the denoiser's ffxDispatch. frame selects a slot; the slot recorded kSlots frames ago has retired.
inline void begin(ID3D12Device* device, ID3D12GraphicsCommandList* list, uint32_t frame)
{
if (!enabled() || !ensure(device, list))
return;
State& s = state();
const UINT slot = frame % kSlots;
volatile uint64_t* values = s.values + slot * kStampsPerSlot;
const UINT used = s.used[slot];
if (used > 1 && values[0] != 0 && values[used - 1] >= values[0])
{
// Frames 1-5 carry the cold start; the stats keep warm frames only.
if (frame > 5 + kSlots)
{
for (UINT i = 0; i + 1 < used; ++i)
{
Stat& stat = s.stats[i];
const Record& r = s.records[slot][i];
stat.hash = r.hash, stat.x = r.x, stat.y = r.y, stat.z = r.z, stat.indirect = r.indirect;
stat.barriers = r.barriers, stat.uavBarriers = r.uavBarriers;
stat.sum += static_cast<double>(values[i + 1] - values[i]) / s.ticksPerMs;
++stat.count;
}
if (++s.frames % profile_log_every() == 0)
log_stats();
}
}
for (UINT i = 0; i < kStampsPerSlot; ++i)
values[i] = 0;
s.records[slot].clear();
s.used[slot] = 0;
t_slot = slot;
t_pso = 0;
t_list = list;
t_frame = frame;
t_dispatchIndex = 0;
t_device = device;
s.barriersSinceDispatch = s.uavBarriersSinceDispatch = 0;
stamp(list, 0, 0, 0, 0, false); // the start
}
inline void end(ID3D12GraphicsCommandList* list)
{
State& s = state();
if (t_list != list)
return;
t_list = nullptr;
stamp(list, 0, 0, 0, 0, false); // after the last dispatch
list->ResolveQueryData(s.heap, D3D12_QUERY_TYPE_TIMESTAMP, t_slot * kStampsPerSlot, s.used[t_slot], s.readback,
8ull * t_slot * kStampsPerSlot);
}
// D4R_FSR_RR_DUMP_DIR=<Windows dir>: also copy the denoiser's two outputs of every frame (RGBA16F, rows packed)
// to <dir>/denoised_<frame>_<diffuse|specular>.raw, for checking replacement shaders against the original.
// The copies are written once the frame has retired (4 frames later, or at teardown).
struct Dump
{
uint32_t frame = 0;
ID3D12Resource* buffer = nullptr;
UINT count = 0;
D3D12_PLACED_SUBRESOURCE_FOOTPRINT footprint[8] = {};
UINT rows[8] = {};
UINT64 rowBytes[8] = {};
const char* names[8] = {};
bool pending = false;
};
inline std::vector<Dump>& dumps(int set)
{
static std::vector<Dump> d[2] = {std::vector<Dump>(8), std::vector<Dump>(8)};
return d[set];
}
inline void dump_write(Dump& d)
{
if (!d.pending)
return;
d.pending = false;
static const std::string dir = env_string("D4R_FSR_RR_DUMP_DIR");
void* mapped = nullptr;
if (FAILED(d.buffer->Map(0, nullptr, &mapped)))
return;
for (UINT i = 0; i < d.count; ++i)
{
char name[512];
std::snprintf(name, sizeof(name), "%s\\%s_%03u.raw", dir.c_str(), d.names[i], d.frame);
FILE* f = std::fopen(name, "wb");
if (f == nullptr)
continue;
const auto* base = static_cast<const uint8_t*>(mapped) + d.footprint[i].Offset;
for (UINT row = 0; row < d.rows[i]; ++row)
std::fwrite(base + static_cast<size_t>(row) * d.footprint[i].Footprint.RowPitch, 1, d.rowBytes[i], f);
std::fclose(f);
}
d.buffer->Unmap(0, nullptr);
}
// set 0/1: independent rings (e.g. the denoiser's inputs and its outputs). Textures are in `state` and return to it.
inline void dump_record(int set, ID3D12Device* device, ID3D12GraphicsCommandList* list, uint32_t frame, UINT count,
ID3D12Resource* const* textures, const char* const* names, const D3D12_RESOURCE_STATES state)
{
static const bool on = !env_string("D4R_FSR_RR_DUMP_DIR").empty();
if (!on || count > 8)
return;
Dump& d = dumps(set)[frame % dumps(set).size()];
dump_write(d); // the frame that used this slot has retired
if (d.buffer == nullptr)
{
UINT64 offset = 0;
for (UINT i = 0; i < count; ++i)
{
D3D12_RESOURCE_DESC desc = textures[i]->GetDesc();
UINT64 total = 0;
device->GetCopyableFootprints(&desc, 0, 1, offset, &d.footprint[i], &d.rows[i], &d.rowBytes[i], &total);
offset = (offset + total + 65535) & ~65535ull;
}
D3D12_HEAP_PROPERTIES heap = {};
heap.Type = D3D12_HEAP_TYPE_READBACK;
D3D12_RESOURCE_DESC buffer = {};
buffer.Dimension = D3D12_RESOURCE_DIMENSION_BUFFER;
buffer.Width = offset;
buffer.Height = 1;
buffer.DepthOrArraySize = 1;
buffer.MipLevels = 1;
buffer.SampleDesc.Count = 1;
buffer.Layout = D3D12_TEXTURE_LAYOUT_ROW_MAJOR;
if (FAILED(device->CreateCommittedResource(&heap, D3D12_HEAP_FLAG_NONE, &buffer, D3D12_RESOURCE_STATE_COPY_DEST,
nullptr, __uuidof(ID3D12Resource), reinterpret_cast<void**>(&d.buffer))))
{
d.buffer = nullptr;
return;
}
}
d.count = count;
D3D12_RESOURCE_BARRIER barriers[8] = {};
for (UINT i = 0; i < count; ++i)
{
d.names[i] = names[i];
barriers[i].Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
barriers[i].Transition.pResource = textures[i];
barriers[i].Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
barriers[i].Transition.StateBefore = state;
barriers[i].Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_SOURCE;
}
list->ResourceBarrier(count, barriers);
for (UINT i = 0; i < count; ++i)
{
D3D12_TEXTURE_COPY_LOCATION dst = {}, src = {};
dst.pResource = d.buffer;
dst.Type = D3D12_TEXTURE_COPY_TYPE_PLACED_FOOTPRINT;
dst.PlacedFootprint = d.footprint[i];
src.pResource = textures[i];
src.Type = D3D12_TEXTURE_COPY_TYPE_SUBRESOURCE_INDEX;
list->CopyTextureRegion(&dst, 0, 0, 0, &src, nullptr);
}
for (UINT i = 0; i < count; ++i)
std::swap(barriers[i].Transition.StateBefore, barriers[i].Transition.StateAfter);
list->ResourceBarrier(count, barriers);
d.frame = frame;
d.pending = true;
}
inline void snapshot_after_dispatch(ID3D12GraphicsCommandList* list)
{
const UINT index = ++t_dispatchIndex; // 1-based: matches the D4R_FSR_RR_DISPATCH table (row 0 = start stamp)
static const uint32_t frame = env_uint("D4R_FSR_RR_DP_SNAPSHOT_FRAME", 1);
const auto& points = snapshot_points();
if (t_frame != frame || std::find(points.begin(), points.end(), index) == points.end())
return;
D3D12_RESOURCE_BARRIER uav = {};
uav.Type = D3D12_RESOURCE_BARRIER_TYPE_UAV;
state().resourceBarrier(list, 1, &uav);
Snapshot snap{index, {}};
// the dispatch's root CBV, read through the CPU mapping of whichever recorded buffer holds it
for (size_t i = 0; i < created().size() && t_cbv != 0; ++i)
{
const Created& c = created()[i];
if (c.desc.Dimension != D3D12_RESOURCE_DIMENSION_BUFFER)
continue;
const D3D12_GPU_VIRTUAL_ADDRESS va = c.resource->GetGPUVirtualAddress();
if (t_cbv < va || t_cbv + 1376 > va + c.desc.Width)
continue;
void* mapped = nullptr;
D3D12_RANGE range = {static_cast<SIZE_T>(t_cbv - va), static_cast<SIZE_T>(t_cbv - va + 1376)};
if (SUCCEEDED(c.resource->Map(0, &range, &mapped)))
{
static const std::string dir = env_string("D4R_FSR_RR_DUMP_DIR");
char name[512];
std::snprintf(name, sizeof(name), "%s\\snap_%03u_cbv.raw", dir.c_str(), index);
if (FILE* f = std::fopen(name, "wb"))
{
std::fwrite(static_cast<const uint8_t*>(mapped) + (t_cbv - va), 1, 1376, f);
std::fclose(f);
}
D3D12_RANGE none = {0, 0};
c.resource->Unmap(0, &none);
logf("D4R_FSR_RR_DP_SNAPSHOT: dispatch %u CBV in resource %zu at +%llu", index, i,
static_cast<unsigned long long>(t_cbv - va));
}
break;
}
// CPU-visible buffers (upload copies, e.g. the weights): read through their mapping
for (size_t i = 0; i < created().size(); ++i)
{
const Created& c = created()[i];
D3D12_HEAP_PROPERTIES props = {};
D3D12_HEAP_FLAGS flags = {};
if (c.desc.Dimension != D3D12_RESOURCE_DIMENSION_BUFFER || FAILED(c.resource->GetHeapProperties(&props, &flags)) ||
props.Type != D3D12_HEAP_TYPE_UPLOAD)
continue;
void* mapped = nullptr;
if (SUCCEEDED(c.resource->Map(0, nullptr, &mapped)))
{
static const std::string dir = env_string("D4R_FSR_RR_DUMP_DIR");
char name[512];
std::snprintf(name, sizeof(name), "%s\\snap_%03u_upload%02zu.raw", dir.c_str(), index, i);
if (FILE* f = std::fopen(name, "wb"))
{
std::fwrite(mapped, 1, c.desc.Width, f);
std::fclose(f);
}
D3D12_RANGE none = {0, 0};
c.resource->Unmap(0, &none);
}
}
if (t_cbv != 0)
logf("D4R_FSR_RR_DP_SNAPSHOT: dispatch %u root CBV va 0x%llx", index, static_cast<unsigned long long>(t_cbv));
for (size_t i = 0; i < created().size(); ++i)
{
const Created& c = created()[i];
if (c.desc.Dimension != D3D12_RESOURCE_DIMENSION_BUFFER ||
(c.desc.Flags & D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS) == 0)
continue;
D3D12_HEAP_PROPERTIES heap = {};
heap.Type = D3D12_HEAP_TYPE_READBACK;
D3D12_RESOURCE_DESC buffer = c.desc;
buffer.Flags = D3D12_RESOURCE_FLAG_NONE;
ID3D12Resource* readback = nullptr;
if (FAILED(t_device->CreateCommittedResource(&heap, D3D12_HEAP_FLAG_NONE, &buffer, D3D12_RESOURCE_STATE_COPY_DEST,
nullptr, __uuidof(ID3D12Resource), reinterpret_cast<void**>(&readback))))
continue;
D3D12_RESOURCE_BARRIER t = {};
t.Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
t.Transition.pResource = c.resource;
t.Transition.StateBefore = D3D12_RESOURCE_STATE_UNORDERED_ACCESS;
t.Transition.StateAfter = D3D12_RESOURCE_STATE_COPY_SOURCE;
state().resourceBarrier(list, 1, &t);
list->CopyBufferRegion(readback, 0, c.resource, 0, c.desc.Width);
std::swap(t.Transition.StateBefore, t.Transition.StateAfter);
state().resourceBarrier(list, 1, &t);
snap.copies.push_back({i, readback});
}
snapshots().push_back(std::move(snap));
}
inline void snapshot_flush()
{
static const std::string dir = env_string("D4R_FSR_RR_DUMP_DIR");
for (Snapshot& snap : snapshots())
{
for (auto& [index, readback] : snap.copies)
{
void* mapped = nullptr;
if (SUCCEEDED(readback->Map(0, nullptr, &mapped)))
{
char name[512];
std::snprintf(name, sizeof(name), "%s\\snap_%03u_res%02zu.raw", dir.c_str(), snap.dispatch, index);
if (FILE* f = std::fopen(name, "wb"))
{
std::fwrite(mapped, 1, created()[index].desc.Width, f);
std::fclose(f);
}
readback->Unmap(0, nullptr);
}
readback->Release();
}
logf("D4R_FSR_RR_DP_SNAPSHOT: dispatch %u written (%zu buffers)", snap.dispatch, snap.copies.size());
}
snapshots().clear();
}
inline void dump_flush()
{
snapshot_flush();
for (int set = 0; set < 2; ++set)
for (Dump& d : dumps(set))
dump_write(d);
}
} // namespace fsr_rr_dp
tools/d4r_fsr_rr_identity.h
+172
View File
@@ -0,0 +1,172 @@
// AMD's FSR Ray Regeneration runtime in a process that reports an NVIDIA GPU to the game.
//
// d4r's DLSS games run with WINE_HIDE_AMD_GPU (PROTON_FORCE_NVAPI): Vulkan, and so DXGI and
// vkd3d-proton, report vendor 0x10de, which is what lets the game offer DLSS at all. vkd3d-proton
// loads Proton's amdxc64.dll - the shim standing in for AMD's D3D12 driver-extension DLL, which
// forwards to vkd3d-proton's own extensions - only when Vulkan reports vendor 0x1002
// (d3d12_device_init_vendor_hacks), so under the spoof it never does. AMD's runtimes look that module
// up by name, and without it the unmodified denoiser 1.2.0 DLL enumerates no provider for the same
// RX 7700 XT it serves without the spoof (tools/fsr_rr_probe.cpp, 2026-10-13).
//
// For a GPU that sysfs says is AMD's, the module is loaded here exactly as vkd3d-proton itself would
// have. The game's view of the GPU is not changed, and nothing of AMD's is patched. The real PCI ids
// are read from sysfs (Z:\sys\bus\pci\devices\...) at the address Vulkan reports for vkd3d-proton's
// physical device, which the vendor-id spoof does not touch: nothing is assumed without that evidence.
#pragma once
#include <windows.h>
#include <d3d12.h>
#include <dxgi1_4.h>
#include <vulkan/vulkan_core.h>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <string>
#include "d4r_vkd3d_interop.h"
struct D4rPciIdentity
{
uint32_t vendor = 0, device = 0;
bool known = false;
};
// The identity the D3D12 device's own DXGI adapter reports (spoofed or not).
inline D4rPciIdentity d4r_dxgi_identity(ID3D12Device* device)
{
D4rPciIdentity identity;
LUID luid;
device->GetAdapterLuid(&luid);
IDXGIFactory4* factory = nullptr;
if (FAILED(CreateDXGIFactory2(0, __uuidof(IDXGIFactory4), reinterpret_cast<void**>(&factory))))
return identity;
IDXGIAdapter1* adapter = nullptr;
if (SUCCEEDED(factory->EnumAdapterByLuid(luid, __uuidof(IDXGIAdapter1), reinterpret_cast<void**>(&adapter))))
{
DXGI_ADAPTER_DESC1 desc = {};
if (SUCCEEDED(adapter->GetDesc1(&desc)))
identity = {desc.VendorId, desc.DeviceId, true};
adapter->Release();
}
factory->Release();
return identity;
}
inline bool d4r_read_hex_file(const char* path, uint32_t& value)
{
FILE* file = std::fopen(path, "rb");
if (file == nullptr)
return false;
char text[32] = {};
const size_t length = std::fread(text, 1, sizeof(text) - 1, file);
std::fclose(file);
if (length == 0)
return false;
char* end = nullptr;
const unsigned long parsed = std::strtoul(text, &end, 16);
if (end == text)
return false;
value = static_cast<uint32_t>(parsed);
return true;
}
// The real PCI identity of the GPU behind a vkd3d-proton device: Vulkan's PCI bus address, then that
// device's sysfs vendor/device files.
inline D4rPciIdentity d4r_true_identity(ID3D12Device* device, std::string& detail)
{
D4rPciIdentity identity;
ID3D12DXVKInteropDevice1* interop = nullptr;
if (FAILED(device->QueryInterface(__uuidof(ID3D12DXVKInteropDevice1), reinterpret_cast<void**>(&interop))))
{
detail = "not a vkd3d-proton device";
return identity;
}
VkInstance instance = VK_NULL_HANDLE;
VkPhysicalDevice physical = VK_NULL_HANDLE;
VkDevice vkDevice = VK_NULL_HANDLE;
const HRESULT handles = interop->GetVulkanHandles(&instance, &physical, &vkDevice);
interop->Release();
HMODULE vulkan = GetModuleHandleA("vulkan-1.dll");
if (vulkan == nullptr)
vulkan = LoadLibraryA("vulkan-1.dll");
auto instanceProc = vulkan != nullptr ? reinterpret_cast<PFN_vkGetInstanceProcAddr>(
reinterpret_cast<void*>(GetProcAddress(vulkan, "vkGetInstanceProcAddr")))
: nullptr;
auto properties2 = instanceProc != nullptr && SUCCEEDED(handles)
? reinterpret_cast<PFN_vkGetPhysicalDeviceProperties2>(
instanceProc(instance, "vkGetPhysicalDeviceProperties2"))
: nullptr;
if (properties2 == nullptr)
{
detail = "no Vulkan physical-device query";
return identity;
}
VkPhysicalDevicePCIBusInfoPropertiesEXT bus = {};
bus.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PCI_BUS_INFO_PROPERTIES_EXT;
VkPhysicalDeviceDriverProperties driver = {};
driver.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DRIVER_PROPERTIES;
driver.pNext = &bus;
VkPhysicalDeviceProperties2 props = {};
props.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2;
props.pNext = &driver;
properties2(physical, &props);
if (bus.pciBus == 0 && bus.pciDevice == 0 && bus.pciDomain == 0 && bus.pciFunction == 0)
{
detail = "Vulkan reports no PCI bus address";
return identity;
}
char base[96];
std::snprintf(base, sizeof(base), "Z:\\sys\\bus\\pci\\devices\\%04x:%02x:%02x.%x\\", bus.pciDomain, bus.pciBus,
bus.pciDevice, bus.pciFunction);
const std::string vendorPath = std::string(base) + "vendor", devicePath = std::string(base) + "device";
if (!d4r_read_hex_file(vendorPath.c_str(), identity.vendor) || !d4r_read_hex_file(devicePath.c_str(), identity.device))
{
detail = "cannot read " + vendorPath;
return identity;
}
identity.known = true;
char text[256];
std::snprintf(text, sizeof(text), "PCI %04x:%02x:%02x.%x is %04x:%04x (Vulkan reports %04x:%04x, driver %s)",
bus.pciDomain, bus.pciBus, bus.pciDevice, bus.pciFunction, identity.vendor, identity.device,
props.properties.vendorID, props.properties.deviceID, driver.driverName);
detail = text;
return identity;
}
// Makes Proton's amdxc64.dll available to AMD's runtime when the GPU really is AMD's. Returns false
// only when it should be there and could not be loaded; `detail` says what was found either way.
inline bool d4r_fsr_rr_driver_extension(ID3D12Device* device, std::string& detail)
{
const D4rPciIdentity reported = d4r_dxgi_identity(device);
std::string pci;
const D4rPciIdentity real = d4r_true_identity(device, pci);
char prefix[512];
std::snprintf(prefix, sizeof(prefix), "DXGI reports %04x:%04x; %s", reported.vendor, reported.device,
real.known ? pci.c_str() : ("real PCI identity unknown (" + pci + ")").c_str());
detail = prefix;
if (!real.known || real.vendor != 0x1002)
{
detail += "; not an AMD GPU by sysfs: amdxc64.dll left alone";
return true;
}
if (GetModuleHandleA("winevulkan.dll") == nullptr)
{
// A native Windows driver's amdxc64.dll is AMD's own and is not loaded from here.
detail += "; not running on winevulkan: amdxc64.dll left alone";
return true;
}
HMODULE extension = GetModuleHandleA("amdxc64.dll");
const bool already = extension != nullptr;
if (extension == nullptr)
extension = LoadLibraryA("amdxc64.dll");
if (extension == nullptr)
{
detail += "; amdxc64.dll (Proton's AMD driver-extension shim) is not available";
return false;
}
char path[MAX_PATH] = {};
GetModuleFileNameA(extension, path, MAX_PATH);
detail += std::string(already ? "; amdxc64.dll already loaded (" : "; loaded amdxc64.dll (") + path + ")";
return true;
}
tools/d4r_fsr_rr_passes.h
+239
View File
@@ -0,0 +1,239 @@
// The FSR Ray Regeneration backend's own two compute passes (kernels/support/fsr_rr_prepare.comp and
// fsr_rr_compose.comp), recorded into the game's D3D12 command list through vkd3d-proton's Vulkan
// interop. Descriptors are pushed (VK_KHR_push_descriptor, which vkd3d-proton requires), so nothing
// here outlives the command buffer that recorded it except the pipelines.
#pragma once
#include <vulkan/vulkan_core.h>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include "d4r_vulkan_staging_spv.h"
struct D4rFsrRrPrepareConstants
{
float clipToView[16];
float viewToPreviousView[16];
float renderSize[4];
float motion[4];
uint32_t colorDepthBase[4];
uint32_t motionBaseFlags[4];
float depthParams[4];
float detail[4];
};
static_assert(sizeof(D4rFsrRrPrepareConstants) == 224, "fsr_rr_prepare.comp push constants");
enum D4rFsrRrPrepareFlags : uint32_t
{
kFsrRrHardwareDepth = 1u,
kFsrRrRoughnessPacked = 2u,
kFsrRrSpecularHit = 4u,
kFsrRrSpecularHitInW = 8u,
kFsrRrDiffuseHit = 16u,
kFsrRrDiffuseHitInW = 32u,
kFsrRrNormalsUnorm = 64u,
kFsrRrBeforeParticles = 128u,
kFsrRrInvalidNormalPassthrough = 256u,
};
// One image a pass reads or writes: the view and the layout the image is in while the pass runs.
struct D4rFsrRrImage
{
VkImageView view = VK_NULL_HANDLE;
VkImageLayout layout = VK_IMAGE_LAYOUT_GENERAL;
};
struct D4rFsrRrPasses
{
static constexpr uint32_t kPrepareInputs = 10, kPrepareOutputs = 8, kComposeImages = 8;
VkDevice device = VK_NULL_HANDLE;
VkSampler sampler = VK_NULL_HANDLE;
VkDescriptorSetLayout prepareSet = VK_NULL_HANDLE, composeSet = VK_NULL_HANDLE;
VkPipelineLayout prepareLayout = VK_NULL_HANDLE, composeLayout = VK_NULL_HANDLE;
VkPipeline preparePipeline = VK_NULL_HANDLE, composePipeline = VK_NULL_HANDLE;
PFN_vkCreateSampler createSampler = nullptr;
PFN_vkDestroySampler destroySampler = nullptr;
PFN_vkCreateDescriptorSetLayout createSetLayout = nullptr;
PFN_vkDestroyDescriptorSetLayout destroySetLayout = nullptr;
PFN_vkCreatePipelineLayout createLayout = nullptr;
PFN_vkDestroyPipelineLayout destroyLayout = nullptr;
PFN_vkCreateShaderModule createShader = nullptr;
PFN_vkDestroyShaderModule destroyShader = nullptr;
PFN_vkCreateComputePipelines createPipelines = nullptr;
PFN_vkDestroyPipeline destroyPipeline = nullptr;
PFN_vkCmdBindPipeline bindPipeline = nullptr;
PFN_vkCmdPushConstants push = nullptr;
PFN_vkCmdPushDescriptorSetKHR pushDescriptors = nullptr;
PFN_vkCmdDispatch dispatch = nullptr;
PFN_vkCmdPipelineBarrier barrier = nullptr;
PFN_vkCreateImageView createView = nullptr;
PFN_vkDestroyImageView destroyView = nullptr;
bool pipeline(const uint32_t* code, size_t bytes, VkPipelineLayout layout, VkPipeline& result)
{
VkShaderModuleCreateInfo shader = {VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO};
shader.codeSize = bytes;
shader.pCode = code;
VkShaderModule module = VK_NULL_HANDLE;
if (createShader(device, &shader, nullptr, &module) != VK_SUCCESS)
return false;
VkComputePipelineCreateInfo info = {VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO};
info.stage = {VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO};
info.stage.stage = VK_SHADER_STAGE_COMPUTE_BIT;
info.stage.module = module;
info.stage.pName = "main";
info.layout = layout;
const VkResult status = createPipelines(device, VK_NULL_HANDLE, 1, &info, nullptr, &result);
destroyShader(device, module, nullptr);
return status == VK_SUCCESS;
}
bool set_layout(const VkDescriptorType* types, uint32_t count, uint32_t pushBytes, VkDescriptorSetLayout& set,
VkPipelineLayout& layout)
{
VkDescriptorSetLayoutBinding bindings[kPrepareInputs + kPrepareOutputs] = {};
for (uint32_t i = 0; i < count; ++i)
{
bindings[i].binding = i;
bindings[i].descriptorType = types[i];
bindings[i].descriptorCount = 1;
bindings[i].stageFlags = VK_SHADER_STAGE_COMPUTE_BIT;
bindings[i].pImmutableSamplers = types[i] == VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER ? &sampler : nullptr;
}
VkDescriptorSetLayoutCreateInfo info = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO};
info.flags = VK_DESCRIPTOR_SET_LAYOUT_CREATE_PUSH_DESCRIPTOR_BIT_KHR;
info.bindingCount = count;
info.pBindings = bindings;
if (createSetLayout(device, &info, nullptr, &set) != VK_SUCCESS)
return false;
VkPushConstantRange range = {VK_SHADER_STAGE_COMPUTE_BIT, 0, pushBytes};
VkPipelineLayoutCreateInfo pipelineLayout = {VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO};
pipelineLayout.setLayoutCount = 1;
pipelineLayout.pSetLayouts = &set;
pipelineLayout.pushConstantRangeCount = 1;
pipelineLayout.pPushConstantRanges = &range;
return createLayout(device, &pipelineLayout, nullptr, &layout) == VK_SUCCESS;
}
// `maxPushConstants`: VkPhysicalDeviceLimits::maxPushConstantsSize (the prepare pass needs 224).
bool init(VkDevice d, PFN_vkGetDeviceProcAddr proc, uint32_t maxPushConstants)
{
device = d;
if (maxPushConstants < sizeof(D4rFsrRrPrepareConstants))
return false;
#define D4R_LOAD(member, name) \
member = reinterpret_cast<decltype(member)>(proc(d, "vk" #name)); \
if (!member) \
return false
D4R_LOAD(createSampler, CreateSampler);
D4R_LOAD(destroySampler, DestroySampler);
D4R_LOAD(createSetLayout, CreateDescriptorSetLayout);
D4R_LOAD(destroySetLayout, DestroyDescriptorSetLayout);
D4R_LOAD(createLayout, CreatePipelineLayout);
D4R_LOAD(destroyLayout, DestroyPipelineLayout);
D4R_LOAD(createShader, CreateShaderModule);
D4R_LOAD(destroyShader, DestroyShaderModule);
D4R_LOAD(createPipelines, CreateComputePipelines);
D4R_LOAD(destroyPipeline, DestroyPipeline);
D4R_LOAD(bindPipeline, CmdBindPipeline);
D4R_LOAD(push, CmdPushConstants);
D4R_LOAD(pushDescriptors, CmdPushDescriptorSetKHR);
D4R_LOAD(dispatch, CmdDispatch);
D4R_LOAD(barrier, CmdPipelineBarrier);
D4R_LOAD(createView, CreateImageView);
D4R_LOAD(destroyView, DestroyImageView);
#undef D4R_LOAD
VkSamplerCreateInfo nearest = {VK_STRUCTURE_TYPE_SAMPLER_CREATE_INFO};
nearest.magFilter = nearest.minFilter = VK_FILTER_NEAREST;
nearest.mipmapMode = VK_SAMPLER_MIPMAP_MODE_NEAREST;
nearest.addressModeU = nearest.addressModeV = nearest.addressModeW = VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE;
if (createSampler(d, &nearest, nullptr, &sampler) != VK_SUCCESS)
return false;
VkDescriptorType prepare[kPrepareInputs + kPrepareOutputs];
for (uint32_t i = 0; i < kPrepareInputs + kPrepareOutputs; ++i)
prepare[i] = i < kPrepareInputs ? VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER : VK_DESCRIPTOR_TYPE_STORAGE_IMAGE;
VkDescriptorType compose[kComposeImages];
for (VkDescriptorType& type : compose)
type = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE;
return set_layout(prepare, kPrepareInputs + kPrepareOutputs, sizeof(D4rFsrRrPrepareConstants), prepareSet,
prepareLayout) &&
set_layout(compose, kComposeImages, 16, composeSet, composeLayout) &&
pipeline(d4r_vk_fsr_rr_prepare, sizeof(d4r_vk_fsr_rr_prepare), prepareLayout, preparePipeline) &&
pipeline(d4r_vk_fsr_rr_compose, sizeof(d4r_vk_fsr_rr_compose), composeLayout, composePipeline);
}
void destroy()
{
if (device == VK_NULL_HANDLE || destroyPipeline == nullptr)
return;
destroyPipeline(device, preparePipeline, nullptr);
destroyPipeline(device, composePipeline, nullptr);
destroyLayout(device, prepareLayout, nullptr);
destroyLayout(device, composeLayout, nullptr);
destroySetLayout(device, prepareSet, nullptr);
destroySetLayout(device, composeSet, nullptr);
destroySampler(device, sampler, nullptr);
*this = D4rFsrRrPasses{};
}
// Everything the D3D12 side wrote before is visible to the pass, and the pass's writes to what
// the D3D12 side records after.
void full_barrier(VkCommandBuffer cmd)
{
VkMemoryBarrier memory = {VK_STRUCTURE_TYPE_MEMORY_BARRIER};
memory.srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT;
memory.dstAccessMask = VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT;
barrier(cmd, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT, 0, 1, &memory, 0, nullptr,
0, nullptr);
}
void record(VkCommandBuffer cmd, VkPipeline pipeline, VkPipelineLayout layout, const D4rFsrRrImage* images,
uint32_t count, uint32_t sampledCount, const void* constants, uint32_t constantBytes, uint32_t width,
uint32_t height)
{
VkDescriptorImageInfo infos[kPrepareInputs + kPrepareOutputs] = {};
VkWriteDescriptorSet writes[kPrepareInputs + kPrepareOutputs] = {};
for (uint32_t i = 0; i < count; ++i)
{
infos[i].sampler = i < sampledCount ? sampler : VK_NULL_HANDLE;
infos[i].imageView = images[i].view;
infos[i].imageLayout = images[i].layout;
writes[i].sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
writes[i].dstBinding = i;
writes[i].descriptorCount = 1;
writes[i].descriptorType =
i < sampledCount ? VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER : VK_DESCRIPTOR_TYPE_STORAGE_IMAGE;
writes[i].pImageInfo = &infos[i];
}
bindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, pipeline);
pushDescriptors(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, layout, 0, count, writes);
push(cmd, layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, constantBytes, constants);
dispatch(cmd, (width + 7) / 8, (height + 7) / 8, 1);
}
// images: the 10 sampled inputs, then the 8 storage outputs (fsr_rr_prepare.comp's bindings)
void record_prepare(VkCommandBuffer cmd, const D4rFsrRrImage* images, const D4rFsrRrPrepareConstants& constants,
uint32_t width, uint32_t height)
{
full_barrier(cmd);
record(cmd, preparePipeline, prepareLayout, images, kPrepareInputs + kPrepareOutputs, kPrepareInputs,
&constants, sizeof(constants), width, height);
full_barrier(cmd);
}
// images: denoised diffuse, denoised specular, diffuse albedo, specular albedo, residual, composed, noisy diffuse,
// noisy specular. haloLimit: fsr_rr_compose.comp's limit (0: off)
void record_compose(VkCommandBuffer cmd, const D4rFsrRrImage* images, uint32_t width, uint32_t height,
float haloLimit = 0.0f)
{
uint32_t limitBits;
std::memcpy(&limitBits, &haloLimit, sizeof(limitBits));
const uint32_t constants[4] = {width, height, limitBits, 0};
full_barrier(cmd);
record(cmd, composePipeline, composeLayout, images, kComposeImages, 0, constants, sizeof(constants), width,
height);
full_barrier(cmd);
}
};
tools/d4r_motion_dilation.h
+7
View File
@@ -11,6 +11,13 @@ typedef struct D4rMotionDilationParams
uint32_t radius, reversed_depth;
} D4rMotionDilationParams;
/* Exact LDS specialization; allocation padding is outside this active grid. */
static inline int d4r_motion_dilation_lds_eligible(const D4rMotionDilationParams* p)
{
return p->motion_width == p->render_width && p->motion_height == p->render_height &&
p->jitter_x == 0 && p->jitter_y == 0 && p->radius >= 1 && p->radius <= 2;
}
/* Low-resolution velocity and depth share the render raster grid. Allocation
padding is not part of that grid; retain the original NGX flags and MV scale.
Display-resolution velocity still needs the jittered depth-grid mapping. */
tools/d4r_native_motion.h
+94
View File
@@ -0,0 +1,94 @@
#pragma once
#include <stdint.h>
typedef struct D4rNativeMotionParams
{
uint64_t depth,geometry,scene,matrix,output;
uint64_t depth_pitch,geometry_pitch,output_pitch;
uint32_t width,height,render_width,render_height;
} D4rNativeMotionParams;
#ifdef __cplusplus
static_assert(sizeof(D4rNativeMotionParams)==80);
#endif
/* Native AW2 hardware-depth contract: immutable input planes; disjoint output. */
static inline int d4r_native_motion_parameters_valid(const D4rNativeMotionParams* p)
{
uint64_t pointers[5],sizes[5],pitches[3],row;
unsigned int i;
if(!p || !p->width || !p->height || p->width>4096 || p->height>4096 ||
!p->render_width || !p->render_height || p->render_width>p->width || p->render_height>p->height) return 0;
row=(uint64_t)p->width*4;
pitches[0]=p->depth_pitch;pitches[1]=p->geometry_pitch;pitches[2]=p->output_pitch;
for(i=0;i<3;i++)
{
if(pitches[i]<row || pitches[i]%4 || (p->height>1 && pitches[i]>(UINT64_MAX-row)/(p->height-1))) return 0;
}
pointers[0]=p->depth;pointers[1]=p->geometry;pointers[2]=p->scene;pointers[3]=p->matrix;pointers[4]=p->output;
sizes[0]=(p->height-1)*p->depth_pitch+row;sizes[1]=(p->height-1)*p->geometry_pitch+row;
sizes[2]=71*16;sizes[3]=4*16;sizes[4]=(p->height-1)*p->output_pitch+row;
for(i=0;i<5;i++) if(!pointers[i] || pointers[i]%4 || pointers[i]>UINT64_MAX-sizes[i]) return 0;
for(i=0;i<4;i++)
if(pointers[4]<pointers[i]+sizes[i] && pointers[i]<pointers[4]+sizes[4]) return 0;
return 1;
}
// Additive thin-field ABI: depth coverage is jittered, output velocity grid is not.
typedef struct D4rNativeMotionThinParams
{
D4rNativeMotionParams base;
float jitter_x, jitter_y;
} D4rNativeMotionThinParams;
#ifdef __cplusplus
static_assert(sizeof(D4rNativeMotionThinParams)==88);
#endif
// Preserve the game's vectors outside thin native-silhouette footprints.
typedef struct D4rNativeMotionHybridParams
{
D4rNativeMotionThinParams native;
uint64_t game_motion,game_motion_pitch;
} D4rNativeMotionHybridParams;
#ifdef __cplusplus
static_assert(sizeof(D4rNativeMotionHybridParams)==104);
#endif
static inline int d4r_native_motion_hybrid_parameters_valid(const D4rNativeMotionHybridParams* p)
{
if(!p || !d4r_native_motion_parameters_valid(&p->native.base) || !p->game_motion || p->game_motion%4 ||
!(p->native.jitter_x>=-1 && p->native.jitter_x<=1 && p->native.jitter_y>=-1 && p->native.jitter_y<=1)) return 0;
const D4rNativeMotionParams* b=&p->native.base;
const uint64_t row=(uint64_t)b->render_width*4;
if(p->game_motion_pitch<row || p->game_motion_pitch%4 ||
(b->render_height>1 && p->game_motion_pitch>(UINT64_MAX-row)/(b->render_height-1))) return 0;
const uint64_t size=(b->render_height-1)*p->game_motion_pitch+row;
const uint64_t output_size=(b->height-1)*b->output_pitch+(uint64_t)b->width*4;
if(p->game_motion>UINT64_MAX-size || (p->game_motion<b->output+output_size && b->output<p->game_motion+size)) return 0;
return 1;
}
// Packed silhouette mask and per-pixel wire axes, prepared once for this frame.
typedef struct D4rNativeMotionPreparedParams
{
D4rNativeMotionThinParams native;
uint64_t thin_mask,wire_axes;
uint32_t wire_flags,reserved;
} D4rNativeMotionPreparedParams;
#ifdef __cplusplus
static_assert(sizeof(D4rNativeMotionPreparedParams)==112);
#endif
static inline int d4r_native_motion_prepared_parameters_valid(const D4rNativeMotionPreparedParams* p)
{
if(!p || !d4r_native_motion_parameters_valid(&p->native.base) || !p->thin_mask || p->thin_mask%4 ||
!p->wire_axes || p->wire_axes%4 || (p->wire_flags & ~3u) || p->reserved ||
!(p->native.jitter_x>=-1 && p->native.jitter_x<=1 && p->native.jitter_y>=-1 && p->native.jitter_y<=1)) return 0;
const D4rNativeMotionParams* b=&p->native.base;
const uint64_t mask_size=(uint64_t)((b->width+31)/32)*b->height*4,axes_size=(uint64_t)b->width*b->height;
if(p->thin_mask>UINT64_MAX-mask_size || p->wire_axes>UINT64_MAX-axes_size ||
(p->thin_mask<p->wire_axes+axes_size && p->wire_axes<p->thin_mask+mask_size)) return 0;
const uint64_t pointers[5]={b->depth,b->geometry,b->scene,b->matrix,b->output};
const uint64_t sizes[5]={(b->height-1)*b->depth_pitch+(uint64_t)b->width*4,
(b->height-1)*b->geometry_pitch+(uint64_t)b->width*4,1136,64,
(b->height-1)*b->output_pitch+(uint64_t)b->width*4};
for(unsigned i=0;i<5;i++) if((p->thin_mask<pointers[i]+sizes[i] && pointers[i]<p->thin_mask+mask_size) ||
(p->wire_axes<pointers[i]+sizes[i] && pointers[i]<p->wire_axes+axes_size)) return 0;
return 1;
}
tools/d4r_nvngx_shim.cpp
+1488 -112
View File
File diff suppressed because it is too large. Load diff
tools/d4r_vulkan_frontend.inl
+3 -3
View File
@@ -514,7 +514,7 @@ static void native_vk_run(Feature* feature, NativeVkFrame* frame)
const auto downloaded = ProfileClock::now();
if (!destination.cuda) std::memcpy(destination.mapped, feature->outputHost[0].bytes, feature->outputHost[0].size());
feature->completedFrames++;
if (frame->number <= 3 || frame->number % 120 == 0 || profile_enabled())
if (profile_enabled() ? profile_log_frame(frame->number) : (frame->number <= 3 || frame->number % 120 == 0))
logf("D4R_VULKAN_PROFILE frame=%u stage=%.3f upload=%.3f evaluate=%.3f download=%.3f encode=%.3f ms",
frame->number, profile_ms(stageStart, staged), profile_ms(staged, uploaded),
profile_ms(uploaded, evaluatedAt), profile_ms(evaluatedAt, downloaded),
@@ -539,7 +539,7 @@ static void native_vk_run(Feature* feature, NativeVkFrame* frame)
if (g_nativeVk.setEvent(g_nativeVk.device, frame->outputReady) != VK_SUCCESS)
frame->result = NGX_FAIL_PLATFORM_ERROR;
frame->done = true;
if (frame->number <= 3 || frame->number % 120 == 0 || profile_enabled())
if (profile_enabled() ? profile_log_frame(frame->number) : (frame->number <= 3 || frame->number % 120 == 0))
logf("D4R_VULKAN_WAIT frame=%u input_ready=%.3f signal=%.3f total=%.3f ms",
frame->number, profile_ms(waitStarted, inputsReadyAt),
profile_ms(signalStarted, ProfileClock::now()), profile_ms(waitStarted, ProfileClock::now()));
@@ -857,7 +857,7 @@ D4R_EXPORT NgxResult NVSDK_NGX_VULKAN_EvaluateFeature(VkCommandBuffer cmd, const
// write and all earlier commands which reference the staging objects.
g_nativeVk.cmdSetEvent(cmd, frame->consumed, VK_PIPELINE_STAGE_ALL_COMMANDS_BIT);
owner.frames.push_back(frame);
if (frame->number <= 3 || frame->number % 120 == 0 || profile_enabled())
if (profile_enabled() ? profile_log_frame(frame->number) : (frame->number <= 3 || frame->number % 120 == 0))
logf("D4R_VULKAN_API frame=%u reclaim=%.3f allocate=%.3f record=%.3f total=%.3f ms",
frame->number, profile_ms(apiStarted, reapedAt), profile_ms(reapedAt, allocatedAt),
profile_ms(allocatedAt, ProfileClock::now()), profile_ms(apiStarted, ProfileClock::now()));
tools/d4r_wire_history.h
+109
View File
@@ -0,0 +1,109 @@
#pragma once
#include <stdint.h>
#define D4R_WIRE_HISTORY_CONSERVATIVE 1u
#define D4R_WIRE_HISTORY_SKY_ONLY 2u
typedef struct D4rWireHistoryParams
{
uint64_t current, color, native_depth, motion;
uint64_t previous, previous_age, previous_depth;
uint64_t output, output_age, output_depth, input_depth, output_history;
uint64_t color_pitch, depth_pitch, motion_pitch, output_pitch, input_depth_pitch;
uint32_t width, height, render_width, render_height, history_valid;
float jitter_x, jitter_y;
uint32_t reserved;
} D4rWireHistoryParams;
#ifdef __cplusplus
static_assert(sizeof(D4rWireHistoryParams)==168);
#endif
static inline int d4r_wire_history_parameters_valid(const D4rWireHistoryParams* p)
{
if(!p || !p->width || !p->height || p->width>4096 || p->height>4096 ||
p->render_width<12 || p->render_height<12 || p->render_width>p->width || p->render_height>p->height ||
p->history_valid>1 || !(p->jitter_x>=-1 && p->jitter_x<=1 && p->jitter_y>=-1 && p->jitter_y<=1) ||
!p->current || !p->color || !p->native_depth || !p->motion || !p->previous || !p->previous_age ||
!p->previous_depth || !p->output || !p->output_age || !p->output_depth || !p->input_depth || !p->output_history ||
p->color_pitch<(uint64_t)p->render_width*8 || p->input_depth_pitch<(uint64_t)p->render_width*4 || p->depth_pitch<(uint64_t)p->width*4 ||
p->motion_pitch<(uint64_t)p->width*4 || p->output_pitch<(uint64_t)p->width*8 ||
p->current==p->output || p->previous==p->output || p->previous_age==p->output_age ||
p->previous_depth==p->output_depth || p->output_history==p->output ||
p->output_history==p->current || p->output_history==p->previous) return 0;
return 1;
}
// Sparse publication keeps current immutable until the separate apply dispatch.
// Scratch RGB uses the same pitch; patch_mask has one byte per output pixel.
typedef struct D4rWireHistorySparseParams
{
D4rWireHistoryParams base;
uint64_t patch_mask;
} D4rWireHistorySparseParams;
#ifdef __cplusplus
static_assert(sizeof(D4rWireHistorySparseParams)==176);
#endif
static inline int d4r_wire_history_sparse_parameters_valid(const D4rWireHistorySparseParams* p)
{
uint64_t pointers[12],rows[12],pitches[12],heights[12],bytes,end;
unsigned i;
if(!p || !d4r_wire_history_parameters_valid(&p->base) || !p->patch_mask) return 0;
const D4rWireHistoryParams* q=&p->base;
bytes=(uint64_t)q->width*q->height;
if(p->patch_mask>UINT64_MAX-bytes) return 0;
end=p->patch_mask+bytes;
pointers[0]=q->current;pointers[1]=q->color;pointers[2]=q->native_depth;pointers[3]=q->motion;
pointers[4]=q->previous;pointers[5]=q->previous_age;pointers[6]=q->previous_depth;
pointers[7]=q->output;pointers[8]=q->output_age;pointers[9]=q->output_depth;
pointers[10]=q->input_depth;pointers[11]=q->output_history;
for(i=0;i<12;i++) { rows[i]=(uint64_t)q->width*8;pitches[i]=q->output_pitch;heights[i]=q->height; }
rows[1]=(uint64_t)q->render_width*8;pitches[1]=q->color_pitch;heights[1]=q->render_height;
rows[2]=(uint64_t)q->width*4;pitches[2]=q->depth_pitch;
rows[3]=(uint64_t)q->width*4;pitches[3]=q->motion_pitch;
rows[5]=rows[8]=q->width;pitches[5]=pitches[8]=q->width;
rows[6]=rows[9]=(uint64_t)q->width*4;pitches[6]=pitches[9]=(uint64_t)q->width*4;
rows[10]=(uint64_t)q->render_width*4;pitches[10]=q->input_depth_pitch;heights[10]=q->render_height;
for(i=0;i<12;i++)
{
if(heights[i]>1 && pitches[i]>(UINT64_MAX-rows[i])/(heights[i]-1)) return 0;
bytes=(heights[i]-1)*pitches[i]+rows[i];
if(pointers[i]>UINT64_MAX-bytes || (p->patch_mask<pointers[i]+bytes && pointers[i]<end)) return 0;
}
// current becomes the apply destination: every input and history output must
// be disjoint from it, as must the correction scratch allocation.
const uint64_t current_bytes=(q->height-1)*q->output_pitch+(uint64_t)q->width*8;
const uint64_t scratch_bytes=current_bytes;
for(i=1;i<12;i++)
{
bytes=(heights[i]-1)*pitches[i]+rows[i];
if(q->current<pointers[i]+bytes && pointers[i]<q->current+current_bytes) return 0;
if(i!=7 && q->output<pointers[i]+bytes && pointers[i]<q->output+scratch_bytes) return 0;
}
return 1;
}
// Prepared repair preserves rendered output and live history exactly. RGB/depth
// history entries with age 255 are unspecified and must never be read.
typedef struct D4rWireHistoryPreparedParams
{
D4rWireHistorySparseParams sparse;
uint64_t axes;
} D4rWireHistoryPreparedParams;
#ifdef __cplusplus
static_assert(sizeof(D4rWireHistoryPreparedParams)==184);
#endif
static inline int d4r_wire_history_prepared_parameters_valid(const D4rWireHistoryPreparedParams* p)
{
if(!p || !d4r_wire_history_sparse_parameters_valid(&p->sparse) || !p->axes || (p->sparse.base.reserved & ~3u)) return 0;
const D4rWireHistoryParams* b=&p->sparse.base;
const uint64_t size=(uint64_t)b->width*b->height;
if(p->axes>UINT64_MAX-size || (p->axes<p->sparse.patch_mask+size && p->sparse.patch_mask<p->axes+size)) return 0;
const uint64_t pointers[12]={b->current,b->color,b->native_depth,b->motion,b->previous,b->previous_age,
b->previous_depth,b->output,b->output_age,b->output_depth,b->input_depth,b->output_history};
const uint64_t sizes[12]={(b->height-1)*b->output_pitch+(uint64_t)b->width*8,
(b->render_height-1)*b->color_pitch+(uint64_t)b->render_width*8,
(b->height-1)*b->depth_pitch+(uint64_t)b->width*4,(b->height-1)*b->motion_pitch+(uint64_t)b->width*4,
(b->height-1)*b->output_pitch+(uint64_t)b->width*8,size,size*4,
(b->height-1)*b->output_pitch+(uint64_t)b->width*8,size,size*4,
(b->render_height-1)*b->input_depth_pitch+(uint64_t)b->render_width*4,
(b->height-1)*b->output_pitch+(uint64_t)b->width*8};
for(unsigned i=0;i<12;i++) if(p->axes<pointers[i]+sizes[i] && pointers[i]<p->axes+size) return 0;
return 1;
}
tools/fsr_rr_probe.cpp
+695
View File
@@ -0,0 +1,695 @@
// Standalone check that AMD's FSR Ray Regeneration runtime runs on this D3D12 device, separate from
// any game and from the NGX shim. It:
// 1. creates a D3D12 device and reports the adapter identity the runtime will see through DXGI,
// 2. loads the given amd_fidelityfx_denoiser_dx12.dll (unmodified, by path) and enumerates its
// denoiser providers against that device (FFX_API_QUERY_DESC_TYPE_GET_VERSIONS),
// 3. creates a context (this is where the runtime builds its shader pipelines) and asks it which
// provider it got,
// 4. dispatches a synthetic ray-traced scene with per-frame white noise: a static camera, then a
// moving one (with exact 2.5D motion vectors), then a camera cut with FFX_DENOISER_DISPATCH_RESET,
// 5. reads every output back and compares input and output against the noise-free signals, so a
// passthrough or a stale history cannot pass for denoising.
// Every stage prints one "stage=... result=..." line, and OUT_DIR/probe.json collects the lot.
//
// usage: fsr_rr_probe.exe DENOISER_DLL OUT_DIR [--size WxH] [--frames N] [--signals HEX] [--query-only]
// [--no-driver-extension]
// --no-driver-extension: do not load Proton's amdxc64.dll for an AMD GPU hidden as NVIDIA
// (see d4r_fsr_rr_identity.h); reproduces the "no provider" outcome
// --signals: FfxApiDenoiserSignalFlags of the radiance signals to run (default 0x36: direct and
// indirect diffuse and specular, AMD's four-signal configuration; 0x30 = indirect only)
#define WIDL_EXPLICIT_AGGREGATE_RETURNS
#include <windows.h>
#include <d3d12.h>
#include <dxgi1_4.h>
#include <algorithm>
#include <chrono>
#include <cmath>
#include <cstdarg>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
#include <vector>
#include "d4r_fsr_rr.h"
#include "d4r_fsr_rr_identity.h"
#include "fsr_rr_scene.h"
namespace
{
using namespace fsr_scene;
FILE* g_json = nullptr;
bool g_jsonFirst = true;
void json_raw(const char* format, ...)
{
if (g_json == nullptr)
return;
va_list args;
va_start(args, format);
std::vfprintf(g_json, format, args);
va_end(args);
}
// One top-level "key": value pair; value is preformatted JSON.
void json_field(const char* key, const std::string& value)
{
json_raw("%s\n \"%s\": %s", g_jsonFirst ? "" : ",", key, value.c_str());
g_jsonFirst = false;
}
std::string quoted(const std::string& text)
{
std::string out = "\"";
for (char c : text)
{
if (c == '"' || c == '\\')
out += '\\';
if (static_cast<unsigned char>(c) < 0x20)
continue;
out += c;
}
return out + "\"";
}
std::string narrow(const wchar_t* text)
{
char buffer[1024] = {};
WideCharToMultiByte(CP_UTF8, 0, text, -1, buffer, sizeof(buffer), nullptr, nullptr);
return buffer;
}
std::string format(const char* pattern, ...)
{
char buffer[2048];
va_list args;
va_start(args, pattern);
std::vsnprintf(buffer, sizeof(buffer), pattern, args);
va_end(args);
return buffer;
}
void stage(const char* name, bool ok, const std::string& detail)
{
std::printf("stage=%s result=%s %s\n", name, ok ? "ok" : "FAIL", detail.c_str());
std::fflush(stdout);
json_field((std::string("stage_") + name).c_str(),
format("{\"ok\": %s, \"detail\": %s}", ok ? "true" : "false", quoted(detail).c_str()));
}
std::vector<std::string> g_messages;
void ffx_message(uint32_t type, const wchar_t* message)
{
const std::string text = narrow(message);
std::printf("ffx %s: %s\n", type == FFX_API_MESSAGE_TYPE_WARNING ? "warning" : "error", text.c_str());
std::fflush(stdout);
if (g_messages.size() < 64)
g_messages.push_back(text);
}
// --- D3D12 plumbing -------------------------------------------------------------------
#define CHECK_HR(expr) \
do \
{ \
const HRESULT hr_ = (expr); \
if (FAILED(hr_)) \
{ \
std::printf("%s failed: 0x%08lx (line %d)\n", #expr, static_cast<unsigned long>(hr_), __LINE__); \
return false; \
} \
} while (0)
struct Texture
{
ID3D12Resource* resource = nullptr;
D3D12_RESOURCE_DESC desc = {};
D3D12_PLACED_SUBRESOURCE_FOOTPRINT footprint = {};
UINT rows = 0;
UINT64 rowBytes = 0, total = 0;
D3D12_RESOURCE_STATES state = D3D12_RESOURCE_STATE_COMMON;
};
ID3D12Device* g_device = nullptr;
bool make_texture(Texture& texture, uint32_t width, uint32_t height, DXGI_FORMAT format, bool uav,
D3D12_RESOURCE_STATES state)
{
texture.desc = {};
texture.desc.Dimension = D3D12_RESOURCE_DIMENSION_TEXTURE2D;
texture.desc.Width = width;
texture.desc.Height = height;
texture.desc.DepthOrArraySize = 1;
texture.desc.MipLevels = 1;
texture.desc.Format = format;
texture.desc.SampleDesc.Count = 1;
texture.desc.Flags = uav ? D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS : D3D12_RESOURCE_FLAG_NONE;
D3D12_HEAP_PROPERTIES heap = {};
heap.Type = D3D12_HEAP_TYPE_DEFAULT;
CHECK_HR(g_device->CreateCommittedResource(&heap, D3D12_HEAP_FLAG_NONE, &texture.desc, state, nullptr,
IID_PPV_ARGS(&texture.resource)));
texture.state = state;
g_device->GetCopyableFootprints(&texture.desc, 0, 1, 0, &texture.footprint, &texture.rows, &texture.rowBytes,
&texture.total);
return true;
}
bool make_buffer(ID3D12Resource*& buffer, UINT64 bytes, D3D12_HEAP_TYPE type, uint8_t*& mapped)
{
D3D12_RESOURCE_DESC desc = {};
desc.Dimension = D3D12_RESOURCE_DIMENSION_BUFFER;
desc.Width = bytes;
desc.Height = 1;
desc.DepthOrArraySize = 1;
desc.MipLevels = 1;
desc.SampleDesc.Count = 1;
desc.Layout = D3D12_TEXTURE_LAYOUT_ROW_MAJOR;
D3D12_HEAP_PROPERTIES heap = {};
heap.Type = type;
const D3D12_RESOURCE_STATES state =
type == D3D12_HEAP_TYPE_UPLOAD ? D3D12_RESOURCE_STATE_GENERIC_READ : D3D12_RESOURCE_STATE_COPY_DEST;
CHECK_HR(g_device->CreateCommittedResource(&heap, D3D12_HEAP_FLAG_NONE, &desc, state, nullptr,
IID_PPV_ARGS(&buffer)));
CHECK_HR(buffer->Map(0, nullptr, reinterpret_cast<void**>(&mapped)));
return true;
}
void barrier(ID3D12GraphicsCommandList* list, Texture& texture, D3D12_RESOURCE_STATES after)
{
if (texture.state == after)
return;
D3D12_RESOURCE_BARRIER b = {};
b.Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
b.Transition.pResource = texture.resource;
b.Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
b.Transition.StateBefore = texture.state;
b.Transition.StateAfter = after;
list->ResourceBarrier(1, &b);
texture.state = after;
}
// Copies `texelBytes`-sized texels from `source` (tightly packed rows) into the upload buffer at
// `offset` with the texture's footprint, and records the copy.
void upload(ID3D12GraphicsCommandList* list, Texture& texture, ID3D12Resource* staging, uint8_t* mapped,
UINT64& offset, const void* source, size_t texelBytes)
{
offset = (offset + D3D12_TEXTURE_DATA_PLACEMENT_ALIGNMENT - 1) & ~UINT64(D3D12_TEXTURE_DATA_PLACEMENT_ALIGNMENT - 1);
D3D12_PLACED_SUBRESOURCE_FOOTPRINT footprint = texture.footprint;
footprint.Offset = offset;
const size_t rowBytes = texelBytes * texture.desc.Width;
for (UINT row = 0; row < texture.rows; ++row)
std::memcpy(mapped + offset + row * footprint.Footprint.RowPitch,
static_cast<const uint8_t*>(source) + row * rowBytes, rowBytes);
offset += texture.total;
barrier(list, texture, D3D12_RESOURCE_STATE_COPY_DEST);
D3D12_TEXTURE_COPY_LOCATION dst = {};
dst.pResource = texture.resource;
dst.Type = D3D12_TEXTURE_COPY_TYPE_SUBRESOURCE_INDEX;
D3D12_TEXTURE_COPY_LOCATION src = {};
src.pResource = staging;
src.Type = D3D12_TEXTURE_COPY_TYPE_PLACED_FOOTPRINT;
src.PlacedFootprint = footprint;
list->CopyTextureRegion(&dst, 0, 0, 0, &src, nullptr);
}
void readback(ID3D12GraphicsCommandList* list, Texture& texture, ID3D12Resource* buffer, UINT64 offset)
{
barrier(list, texture, D3D12_RESOURCE_STATE_COPY_SOURCE);
D3D12_PLACED_SUBRESOURCE_FOOTPRINT footprint = texture.footprint;
footprint.Offset = offset;
D3D12_TEXTURE_COPY_LOCATION dst = {};
dst.pResource = buffer;
dst.Type = D3D12_TEXTURE_COPY_TYPE_PLACED_FOOTPRINT;
dst.PlacedFootprint = footprint;
D3D12_TEXTURE_COPY_LOCATION src = {};
src.pResource = texture.resource;
src.Type = D3D12_TEXTURE_COPY_TYPE_SUBRESOURCE_INDEX;
list->CopyTextureRegion(&dst, 0, 0, 0, &src, nullptr);
}
// --- metrics and captures -------------------------------------------------------------
struct Error
{
double input = 0, output = 0, outputVsPrevious = 0, change = 0;
size_t nonFinite = 0;
};
Error measure(const Frame& frame, const Frame* previous, const uint8_t* rows, UINT pitch, uint32_t width,
uint32_t height, int signal)
{
Error e;
size_t n = 0;
for (uint32_t y = 2; y + 2 < height; ++y)
for (uint32_t x = 2; x + 2 < width; ++x)
{
const size_t i = static_cast<size_t>(y) * width + x;
const uint16_t* out = reinterpret_cast<const uint16_t*>(rows + static_cast<size_t>(y) * pitch) + x * 4;
for (int c = 0; c < 3; ++c)
{
const float o = from_half(out[c]);
if (!std::isfinite(o))
{
++e.nonFinite;
continue;
}
if (!frame.valid[i])
continue;
const float clean = frame.clean[signal][i * 3 + c];
const float in = from_half(frame.noisy[signal][i * 4 + c]);
e.input += (in - clean) * (in - clean);
e.output += (o - clean) * (o - clean);
e.change += std::fabs(o - in);
if (previous != nullptr && previous->valid[i])
{
const float old = previous->clean[signal][i * 3 + c];
e.outputVsPrevious += (o - old) * (o - old);
}
++n;
}
}
if (n > 0)
{
e.input = std::sqrt(e.input / n);
e.output = std::sqrt(e.output / n);
e.outputVsPrevious = std::sqrt(e.outputVsPrevious / n);
e.change /= n;
}
return e;
}
// Tonemapped composite (diffuse signals x diffuse albedo + specular signals x specular albedo).
void write_ppm(const std::string& path, uint32_t width, uint32_t height, const std::vector<V3>& radiance)
{
FILE* file = std::fopen(path.c_str(), "wb");
if (file == nullptr)
return;
std::fprintf(file, "P6\n%u %u\n255\n", width, height);
std::vector<uint8_t> row(width * 3);
for (uint32_t y = 0; y < height; ++y)
{
for (uint32_t x = 0; x < width; ++x)
{
const V3 c = radiance[static_cast<size_t>(y) * width + x];
const float rgb[3] = {c.x, c.y, c.z};
for (int k = 0; k < 3; ++k)
{
const float mapped = std::pow(std::max(0.0f, rgb[k]) / (1.0f + std::max(0.0f, rgb[k])), 1.0f / 2.2f);
row[x * 3 + k] = static_cast<uint8_t>(std::lround(saturate(mapped) * 255.0f));
}
}
std::fwrite(row.data(), 1, row.size(), file);
}
std::fclose(file);
}
} // namespace
int wmain(int argc, wchar_t** argv)
{
if (argc < 3)
{
std::printf("usage: fsr_rr_probe.exe DENOISER_DLL OUT_DIR [--size WxH] [--frames N] [--signals HEX] [--query-only]\n");
return 2;
}
const std::wstring dllPath = argv[1];
const std::string outDir = narrow(argv[2]);
uint32_t width = 960, height = 540, frames = 48, signals = 0x36;
bool queryOnly = false, driverExtension = true;
for (int i = 3; i < argc; ++i)
{
const std::wstring arg = argv[i];
if (arg == L"--size" && i + 1 < argc)
std::swscanf(argv[++i], L"%ux%u", &width, &height);
else if (arg == L"--frames" && i + 1 < argc)
frames = static_cast<uint32_t>(std::wcstoul(argv[++i], nullptr, 0));
else if (arg == L"--signals" && i + 1 < argc)
signals = static_cast<uint32_t>(std::wcstoul(argv[++i], nullptr, 16));
else if (arg == L"--query-only")
queryOnly = true;
else if (arg == L"--no-driver-extension")
driverExtension = false;
}
signals &= 0x36;
if (signals == 0 || frames < 12)
{
std::printf("need at least one radiance signal and 12 frames\n");
return 2;
}
CreateDirectoryA(outDir.c_str(), nullptr);
g_json = std::fopen((outDir + "\\probe.json").c_str(), "w");
json_raw("{");
json_field("denoiser_dll", quoted(narrow(dllPath.c_str())));
json_field("render_size", format("[%u, %u]", width, height));
json_field("signal_flags", format("%u", signals));
json_field("frames", format("%u", frames));
int exitCode = 1;
auto finish = [&](int code) {
json_raw("\n}\n");
if (g_json)
std::fclose(g_json);
g_json = nullptr;
return code;
};
// 1. Device, and the adapter identity DXGI reports (what the runtime can see).
IDXGIFactory4* factory = nullptr;
IDXGIAdapter1* adapter = nullptr;
if (FAILED(CreateDXGIFactory1(IID_PPV_ARGS(&factory))) || FAILED(factory->EnumAdapters1(0, &adapter)))
{
stage("device", false, "no DXGI adapter");
return finish(1);
}
DXGI_ADAPTER_DESC1 adapterDesc = {};
adapter->GetDesc1(&adapterDesc);
if (FAILED(D3D12CreateDevice(adapter, D3D_FEATURE_LEVEL_12_0, IID_PPV_ARGS(&g_device))))
{
stage("device", false, "D3D12CreateDevice failed");
return finish(1);
}
D3D12_FEATURE_DATA_SHADER_MODEL shaderModel = {D3D_SHADER_MODEL_6_6};
g_device->CheckFeatureSupport(D3D12_FEATURE_SHADER_MODEL, &shaderModel, sizeof(shaderModel));
stage("device", true,
format("adapter=\"%s\" vendor=0x%04x device=0x%04x luid=%08lx:%08lx shader_model=0x%x",
narrow(adapterDesc.Description).c_str(), adapterDesc.VendorId, adapterDesc.DeviceId,
static_cast<unsigned long>(adapterDesc.AdapterLuid.HighPart),
static_cast<unsigned long>(adapterDesc.AdapterLuid.LowPart),
static_cast<unsigned>(shaderModel.HighestShaderModel)));
json_field("adapter", format("{\"name\": %s, \"vendor\": %u, \"device\": %u}",
quoted(narrow(adapterDesc.Description)).c_str(), adapterDesc.VendorId,
adapterDesc.DeviceId));
// 2. Runtime and provider.
FsrRrRuntime runtime;
if (!runtime.load(dllPath))
{
stage("load", false, format("%s (error %lu)", fsr_rr_stage_name(runtime.stage), GetLastError()));
return finish(1);
}
stage("load", true, "mapped=" + narrow(runtime.mappedPath.c_str()));
if (driverExtension)
{
std::string detail;
const bool loaded = d4r_fsr_rr_driver_extension(g_device, detail);
stage("driver_extension", loaded, detail);
}
const ffxReturnCode_t debug = runtime.set_message_callback(&ffx_message, FFX_API_CONFIGURE_GLOBALDEBUG_LEVEL_VERBOSE);
std::printf("ffx global debug callback: %s\n", fsr_rr_return_name(debug));
const bool provider = runtime.query_providers(g_device);
{
std::string list = "[";
for (size_t i = 0; i < runtime.versionIds.size(); ++i)
list += format("%s{\"id\": \"0x%016llx\", \"name\": %s}", i ? ", " : "",
static_cast<unsigned long long>(runtime.versionIds[i]),
quoted(runtime.versionNames[i]).c_str());
json_field("providers", list + "]");
}
stage("provider", provider,
provider ? format("count=%zu selected=0x%016llx name=\"%s\"", runtime.versionIds.size(),
static_cast<unsigned long long>(runtime.providerId), runtime.providerName.c_str())
: format("%s (query %s)", fsr_rr_stage_name(runtime.stage), fsr_rr_return_name(runtime.queryResult)));
if (!provider || queryOnly)
return finish(provider ? 0 : 1);
FfxApiEffectMemoryUsage memory = {};
const ffxReturnCode_t memoryResult = fsr_rr_query_memory(runtime, g_device, width, height, signals, 0, memory);
json_field("gpu_memory", format("{\"result\": %u, \"total_mb\": %.2f, \"aliasable_mb\": %.2f}", memoryResult,
memory.totalUsageInBytes / 1048576.0, memory.aliasableUsageInBytes / 1048576.0));
std::printf("gpu memory query: %s total=%.2f MB aliasable=%.2f MB\n", fsr_rr_return_name(memoryResult),
memory.totalUsageInBytes / 1048576.0, memory.aliasableUsageInBytes / 1048576.0);
// 3. Context (shader pipelines are created here: the slow and the most likely to fail part).
ffxContext context = nullptr;
const auto createStart = std::chrono::steady_clock::now();
const ffxReturnCode_t created = fsr_rr_create_context(runtime, g_device, width, height, signals, 0, context);
const double createMs =
std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - createStart).count();
json_field("context_create_ms", format("%.1f", createMs));
if (created != FFX_API_RETURN_OK)
{
stage("context", false, format("ffxCreateContext -> %s after %.1f ms", fsr_rr_return_name(created), createMs));
return finish(1);
}
uint64_t actualId = 0;
const std::string actual = fsr_rr_context_provider(runtime, context, &actualId);
stage("context", true, format("%.1f ms, provider in use 0x%016llx \"%s\"", createMs,
static_cast<unsigned long long>(actualId), actual.c_str()));
json_field("provider_in_use", quoted(actual));
// 4. Resources and frames.
ID3D12CommandQueue* queue = nullptr;
D3D12_COMMAND_QUEUE_DESC queueDesc = {};
queueDesc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT;
ID3D12CommandAllocator* allocator = nullptr;
ID3D12GraphicsCommandList* list = nullptr;
ID3D12Fence* fence = nullptr;
if (FAILED(g_device->CreateCommandQueue(&queueDesc, IID_PPV_ARGS(&queue))) ||
FAILED(g_device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_DIRECT, IID_PPV_ARGS(&allocator))) ||
FAILED(g_device->CreateCommandList(0, D3D12_COMMAND_LIST_TYPE_DIRECT, allocator, nullptr, IID_PPV_ARGS(&list))) ||
FAILED(g_device->CreateFence(0, D3D12_FENCE_FLAG_NONE, IID_PPV_ARGS(&fence))))
{
stage("dispatch", false, "command objects");
return finish(1);
}
list->Close();
HANDLE event = CreateEventW(nullptr, FALSE, FALSE, nullptr);
uint64_t fenceValue = 0;
const D3D12_RESOURCE_STATES readState =
D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE | D3D12_RESOURCE_STATE_PIXEL_SHADER_RESOURCE;
Texture depth, motion, normals, diffuseAlbedo, specularAlbedo, input[kSignalCount], output[kSignalCount];
bool ok = make_texture(depth, width, height, DXGI_FORMAT_R32_FLOAT, false, readState) &&
make_texture(motion, width, height, DXGI_FORMAT_R16G16B16A16_FLOAT, false, readState) &&
make_texture(normals, width, height, DXGI_FORMAT_R10G10B10A2_UNORM, false, readState) &&
make_texture(diffuseAlbedo, width, height, DXGI_FORMAT_R8G8B8A8_UNORM, false, readState) &&
make_texture(specularAlbedo, width, height, DXGI_FORMAT_R8G8B8A8_UNORM, false, readState);
for (int s = 0; s < kSignalCount && ok; ++s)
ok = make_texture(input[s], width, height, DXGI_FORMAT_R16G16B16A16_FLOAT, false, readState) &&
make_texture(output[s], width, height, DXGI_FORMAT_R16G16B16A16_FLOAT, true,
D3D12_RESOURCE_STATE_UNORDERED_ACCESS);
ID3D12Resource *staging = nullptr, *readbackBuffer = nullptr, *timestampBuffer = nullptr;
uint8_t *stagingMapped = nullptr, *readbackMapped = nullptr, *timestampMapped = nullptr;
const UINT64 planeBytes = output[0].total;
ok = ok && make_buffer(staging, 16 * (planeBytes + 65536), D3D12_HEAP_TYPE_UPLOAD, stagingMapped) &&
make_buffer(readbackBuffer, kSignalCount * planeBytes, D3D12_HEAP_TYPE_READBACK, readbackMapped) &&
make_buffer(timestampBuffer, 16, D3D12_HEAP_TYPE_READBACK, timestampMapped);
ID3D12QueryHeap* timestamps = nullptr;
D3D12_QUERY_HEAP_DESC queryDesc = {};
queryDesc.Type = D3D12_QUERY_HEAP_TYPE_TIMESTAMP;
queryDesc.Count = 2;
ok = ok && SUCCEEDED(g_device->CreateQueryHeap(&queryDesc, IID_PPV_ARGS(&timestamps)));
UINT64 ticksPerSecond = 0;
queue->GetTimestampFrequency(&ticksPerSecond);
if (!ok)
{
stage("dispatch", false, "resource creation");
return finish(1);
}
const float aspect = static_cast<float>(width) / height;
const M4 projection = projection_matrix(1.0f, aspect, 0.1f, 1000.0f);
const uint32_t cutFrame = frames * 2 / 3, moveStart = frames / 3;
auto camera_at = [&](uint32_t f) -> Camera {
if (f >= cutFrame)
return {{3.5f, 1.6f, 1.5f}, {-0.5f, -0.6f, 7.5f}};
const float t = f >= moveStart ? static_cast<float>(f - moveStart) : 0.0f;
return {{-0.6f + 0.04f * t, 0.6f, -1.5f}, {0.0f + 0.02f * t, -0.1f, 6.5f}};
};
Frame frameData, previousFrame;
std::vector<double> gpuMs, cpuMs;
std::string frameLog = "[";
bool dispatchOk = true, converged = true, resetOk = true, finite = true, notPassthrough = true;
Camera previousCamera = camera_at(0);
for (uint32_t f = 0; f < frames; ++f)
{
const Camera camera = camera_at(f);
const bool reset = f == 0 || f == cutFrame;
std::swap(previousFrame, frameData);
render(frameData, width, height, camera, reset ? camera : previousCamera, projection, f);
allocator->Reset();
list->Reset(allocator, nullptr);
UINT64 offset = 0;
upload(list, depth, staging, stagingMapped, offset, frameData.depth.data(), 4);
upload(list, motion, staging, stagingMapped, offset, frameData.motion.data(), 8);
upload(list, normals, staging, stagingMapped, offset, frameData.normals.data(), 4);
upload(list, diffuseAlbedo, staging, stagingMapped, offset, frameData.diffuseAlbedo.data(), 4);
upload(list, specularAlbedo, staging, stagingMapped, offset, frameData.specularAlbedo.data(), 4);
for (int s = 0; s < kSignalCount; ++s)
if (signals & kSignalFlags[s])
upload(list, input[s], staging, stagingMapped, offset, frameData.noisy[s].data(), 8);
Texture* all[] = {&depth, &motion, &normals, &diffuseAlbedo, &specularAlbedo};
for (Texture* t : all)
barrier(list, *t, readState);
for (int s = 0; s < kSignalCount; ++s)
{
barrier(list, input[s], readState);
barrier(list, output[s], D3D12_RESOURCE_STATE_UNORDERED_ACCESS);
}
ffxDispatchDescDenoiser desc = {};
desc.header.type = FFX_API_DISPATCH_DESC_TYPE_DENOISER;
desc.commandList = list;
const uint32_t readFfx = FFX_API_RESOURCE_STATE_PIXEL_COMPUTE_READ;
desc.linearDepth = fsr_rr_texture(depth.resource, depth.desc, FFX_API_SURFACE_FORMAT_R32_FLOAT, readFfx);
desc.motionVectors = fsr_rr_texture(motion.resource, motion.desc, FFX_API_SURFACE_FORMAT_R16G16B16A16_FLOAT, readFfx);
desc.normals = fsr_rr_texture(normals.resource, normals.desc, FFX_API_SURFACE_FORMAT_R10G10B10A2_UNORM, readFfx);
desc.specularAlbedo = fsr_rr_texture(specularAlbedo.resource, specularAlbedo.desc, FFX_API_SURFACE_FORMAT_R8G8B8A8_UNORM, readFfx);
desc.diffuseAlbedo = fsr_rr_texture(diffuseAlbedo.resource, diffuseAlbedo.desc, FFX_API_SURFACE_FORMAT_R8G8B8A8_UNORM, readFfx);
desc.motionVectorScale = {1.0f, 1.0f, 1.0f};
desc.jitterOffsets = {0.0f, 0.0f};
const V3 delta = previousCamera.eye - camera.eye;
desc.cameraPositionDelta = reset ? FfxApiFloatCoords3D{0, 0, 0} : FfxApiFloatCoords3D{delta.x, delta.y, delta.z};
const M4 view = view_matrix(camera.eye, camera.target);
std::memcpy(&desc.view, &view, sizeof(desc.view));
std::memcpy(&desc.projection, &projection, sizeof(desc.projection));
desc.linearDepthBounds = {0.0f, 1000.0f};
desc.renderSize = {width, height};
desc.frameIndex = f;
desc.flags = reset ? FFX_DENOISER_DISPATCH_RESET : 0;
ffxDispatchDescDenoiserDirectDiffuse dd = {};
ffxDispatchDescDenoiserDirectSpecular ds = {};
ffxDispatchDescDenoiserIndirectDiffuse id = {};
ffxDispatchDescDenoiserIndirectSpecular is = {};
dd.header.type = FFX_API_DISPATCH_DESC_TYPE_DENOISER_DIRECT_DIFFUSE;
ds.header.type = FFX_API_DISPATCH_DESC_TYPE_DENOISER_DIRECT_SPECULAR;
id.header.type = FFX_API_DISPATCH_DESC_TYPE_DENOISER_INDIRECT_DIFFUSE;
is.header.type = FFX_API_DISPATCH_DESC_TYPE_DENOISER_INDIRECT_SPECULAR;
FfxApiDenoiserSignal* signalDescs[kSignalCount] = {&dd.signal, &ds.signal, &id.signal, &is.signal};
ffxApiHeader* headers[kSignalCount] = {&dd.header, &ds.header, &id.header, &is.header};
ffxApiHeader* tail = &desc.header;
for (int s = 0; s < kSignalCount; ++s)
{
if (!(signals & kSignalFlags[s]))
continue;
signalDescs[s]->input = fsr_rr_texture(input[s].resource, input[s].desc, FFX_API_SURFACE_FORMAT_R16G16B16A16_FLOAT, readFfx);
signalDescs[s]->output = fsr_rr_texture(output[s].resource, output[s].desc, FFX_API_SURFACE_FORMAT_R16G16B16A16_FLOAT,
FFX_API_RESOURCE_STATE_UNORDERED_ACCESS);
tail->pNext = headers[s];
tail = headers[s];
}
list->EndQuery(timestamps, D3D12_QUERY_TYPE_TIMESTAMP, 0);
const auto cpuStart = std::chrono::steady_clock::now();
const ffxReturnCode_t dispatched = runtime.dispatch(&context, &desc.header);
cpuMs.push_back(std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - cpuStart).count());
list->EndQuery(timestamps, D3D12_QUERY_TYPE_TIMESTAMP, 1);
list->ResolveQueryData(timestamps, D3D12_QUERY_TYPE_TIMESTAMP, 0, 2, timestampBuffer, 0);
if (dispatched != FFX_API_RETURN_OK)
{
stage("dispatch", false, format("frame %u: ffxDispatch -> %s", f, fsr_rr_return_name(dispatched)));
dispatchOk = false;
list->Close();
break;
}
for (int s = 0; s < kSignalCount; ++s)
if (signals & kSignalFlags[s])
readback(list, output[s], readbackBuffer, s * planeBytes);
if (FAILED(list->Close()))
{
stage("dispatch", false, format("frame %u: Close failed", f));
dispatchOk = false;
break;
}
ID3D12CommandList* lists[] = {list};
queue->ExecuteCommandLists(1, lists);
queue->Signal(fence, ++fenceValue);
fence->SetEventOnCompletion(fenceValue, event);
if (WaitForSingleObject(event, 20000) != WAIT_OBJECT_0)
{
stage("dispatch", false, format("frame %u: GPU did not finish within 20 s", f));
dispatchOk = false;
break;
}
const HRESULT removed = g_device->GetDeviceRemovedReason();
if (FAILED(removed))
{
stage("dispatch", false, format("frame %u: device removed 0x%08lx", f, static_cast<unsigned long>(removed)));
dispatchOk = false;
break;
}
uint64_t stamps[2];
std::memcpy(stamps, timestampMapped, sizeof(stamps));
const double gpu = ticksPerSecond ? (stamps[1] - stamps[0]) * 1000.0 / ticksPerSecond : -1.0;
gpuMs.push_back(gpu);
// 5. Compare against the clean signals.
frameLog += format("%s\n {\"frame\": %u, \"reset\": %s, \"gpu_ms\": %.3f, \"cpu_ms\": %.3f, \"signals\": {",
f ? "," : "", f, reset ? "true" : "false", gpu, cpuMs.back());
bool firstSignal = true;
std::vector<V3> composeOut(static_cast<size_t>(width) * height), composeIn(composeOut.size()),
composeClean(composeOut.size());
for (int s = 0; s < kSignalCount; ++s)
{
if (!(signals & kSignalFlags[s]))
continue;
const uint8_t* rows = readbackMapped + s * planeBytes;
const Error e = measure(frameData, f == cutFrame ? &previousFrame : nullptr, rows,
output[s].footprint.Footprint.RowPitch, width, height, s);
frameLog += format("%s\"%s\": {\"rmse_in\": %.5f, \"rmse_out\": %.5f, \"mean_abs_change\": %.5f, "
"\"rmse_out_vs_previous_scene\": %.5f, \"nonfinite\": %zu}",
firstSignal ? "" : ", ", kSignalNames[s], e.input, e.output, e.change,
e.outputVsPrevious, e.nonFinite);
firstSignal = false;
finite = finite && e.nonFinite == 0;
notPassthrough = notPassthrough && (f < 2 || e.change > 1e-3 * (e.input + 1e-6));
// Converged before the cut: at least 8 static or slowly moving frames into history.
if (f + 1 == cutFrame && !(e.output < 0.5 * e.input))
converged = false;
if (f == cutFrame && !(e.output < e.outputVsPrevious))
resetOk = false;
if (f + 1 == frames && !(e.output < 0.6 * e.input))
resetOk = false;
const bool diffuse = s == kDirectDiffuse || s == kIndirectDiffuse;
for (uint32_t y = 0; y < height; ++y)
for (uint32_t x = 0; x < width; ++x)
{
const size_t i = static_cast<size_t>(y) * width + x;
const V3 albedo = diffuse ? frameData.diffuseLinear[i] : frameData.specularLinear[i];
const uint16_t* o = reinterpret_cast<const uint16_t*>(rows + static_cast<size_t>(y) * output[s].footprint.Footprint.RowPitch) + x * 4;
composeOut[i] = composeOut[i] + mul(V3{from_half(o[0]), from_half(o[1]), from_half(o[2])}, albedo);
composeIn[i] = composeIn[i] + mul(V3{from_half(frameData.noisy[s][i * 4]), from_half(frameData.noisy[s][i * 4 + 1]),
from_half(frameData.noisy[s][i * 4 + 2])}, albedo);
composeClean[i] = composeClean[i] + mul(V3{frameData.clean[s][i * 3], frameData.clean[s][i * 3 + 1],
frameData.clean[s][i * 3 + 2]}, albedo);
}
}
frameLog += "}}";
if (f == 0 || f + 1 == moveStart || f + 1 == cutFrame || f == cutFrame || f == cutFrame + 1 || f + 1 == frames)
{
write_ppm(format("%s\\frame%03u_noisy.ppm", outDir.c_str(), f), width, height, composeIn);
write_ppm(format("%s\\frame%03u_denoised.ppm", outDir.c_str(), f), width, height, composeOut);
write_ppm(format("%s\\frame%03u_clean.ppm", outDir.c_str(), f), width, height, composeClean);
}
previousCamera = camera;
}
json_field("frame_log", frameLog + "\n ]");
if (dispatchOk)
{
std::vector<double> warm(gpuMs.begin() + std::min<size_t>(gpuMs.size(), 3), gpuMs.end());
std::sort(warm.begin(), warm.end());
const double median = warm.empty() ? -1 : warm[warm.size() / 2];
std::vector<double> cpuWarm(cpuMs.begin() + std::min<size_t>(cpuMs.size(), 3), cpuMs.end());
std::sort(cpuWarm.begin(), cpuWarm.end());
stage("dispatch", true, format("%u frames, GPU median %.3f ms (first frame %.3f ms), CPU record median %.3f ms",
frames, median, gpuMs.empty() ? -1.0 : gpuMs[0],
cpuWarm.empty() ? -1.0 : cpuWarm[cpuWarm.size() / 2]));
json_field("gpu_ms_median", format("%.4f", median));
stage("output_finite", finite, finite ? "no NaN/Inf in any output" : "non-finite output values");
stage("not_passthrough", notPassthrough, "outputs differ from the noisy inputs");
stage("denoised", converged, format("before the cut (frame %u) every signal's RMSE vs clean is < 0.5x the input's",
cutFrame - 1));
stage("reset", resetOk, format("frame %u (reset) is closer to the new scene than the old one; final frame < 0.6x input",
cutFrame));
exitCode = finite && notPassthrough && converged && resetOk ? 0 : 1;
}
std::string messages = "[";
for (size_t i = 0; i < g_messages.size(); ++i)
messages += (i ? ", " : "") + quoted(g_messages[i]);
json_field("ffx_messages", messages + "]");
runtime.destroyContext(&context, nullptr);
return finish(exitCode);
}
tools/fsr_rr_scene.h
+515
View File
@@ -0,0 +1,515 @@
// The synthetic ray-traced scene shared by tools/fsr_rr_probe.cpp (AMD's denoiser alone) and
// tools/rr_backend_check.cpp (a game's NGX Ray Reconstruction request through the shim): a camera over
// a checkered floor, three spheres and a wall, with smooth noise-free lighting signals and one-sample
// white-noise versions of them, every guide the denoisers read, and exact 2.5D motion. Because the
// noise-free signals are known, outputs are compared against ground truth rather than another
// denoiser.
#pragma once
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <vector>
namespace fsr_scene
{
// --- small math -----------------------------------------------------------------
struct V3
{
float x, y, z;
};
inline V3 operator+(V3 a, V3 b) { return {a.x + b.x, a.y + b.y, a.z + b.z}; }
inline V3 operator-(V3 a, V3 b) { return {a.x - b.x, a.y - b.y, a.z - b.z}; }
inline V3 operator*(V3 a, float s) { return {a.x * s, a.y * s, a.z * s}; }
inline V3 mul(V3 a, V3 b) { return {a.x * b.x, a.y * b.y, a.z * b.z}; }
inline float dot(V3 a, V3 b) { return a.x * b.x + a.y * b.y + a.z * b.z; }
inline V3 cross(V3 a, V3 b) { return {a.y * b.z - a.z * b.y, a.z * b.x - a.x * b.z, a.x * b.y - a.y * b.x}; }
inline V3 normalize(V3 a) { return a * (1.0f / std::sqrt(dot(a, a))); }
inline float saturate(float v) { return std::min(1.0f, std::max(0.0f, v)); }
inline uint16_t to_half(float value)
{
uint32_t bits;
std::memcpy(&bits, &value, 4);
const uint32_t sign = (bits >> 16) & 0x8000u;
int exponent = static_cast<int>((bits >> 23) & 0xff) - 127 + 15;
uint32_t mantissa = bits & 0x7fffffu;
if (((bits >> 23) & 0xff) == 0xff)
return static_cast<uint16_t>(sign | 0x7c00u | (mantissa ? 0x200u : 0u));
if (exponent >= 31)
return static_cast<uint16_t>(sign | 0x7bffu); // clamp to the largest finite half
if (exponent <= 0)
{
if (exponent < -10)
return static_cast<uint16_t>(sign);
mantissa |= 0x800000u;
const uint32_t shift = static_cast<uint32_t>(14 - exponent);
uint32_t half = mantissa >> shift;
if ((mantissa >> (shift - 1)) & 1u)
++half;
return static_cast<uint16_t>(sign | half);
}
uint32_t half = sign | (static_cast<uint32_t>(exponent) << 10) | (mantissa >> 13);
if (mantissa & 0x1000u)
++half;
return static_cast<uint16_t>(half);
}
inline float from_half(uint16_t half)
{
const uint32_t sign = (half & 0x8000u) << 16;
uint32_t exponent = (half >> 10) & 0x1fu;
uint32_t mantissa = half & 0x3ffu;
uint32_t bits;
if (exponent == 0)
{
if (mantissa == 0)
bits = sign;
else
{
exponent = 127 - 15 + 1;
while ((mantissa & 0x400u) == 0)
{
mantissa <<= 1;
--exponent;
}
bits = sign | (exponent << 23) | ((mantissa & 0x3ffu) << 13);
}
}
else if (exponent == 31)
bits = sign | 0x7f800000u | (mantissa << 13);
else
bits = sign | ((exponent - 15 + 127) << 23) | (mantissa << 13);
float value;
std::memcpy(&value, &bits, 4);
return value;
}
// Row-major, row-vector (v' = vM) 4x4, the convention ffx_denoiser.h asks for.
struct M4
{
float m[4][4];
};
inline M4 view_matrix(V3 eye, V3 target)
{
const V3 forward = normalize(target - eye);
const V3 right = normalize(cross({0, 1, 0}, forward));
const V3 up = cross(forward, right);
M4 v = {};
v.m[0][0] = right.x, v.m[0][1] = up.x, v.m[0][2] = forward.x;
v.m[1][0] = right.y, v.m[1][1] = up.y, v.m[1][2] = forward.y;
v.m[2][0] = right.z, v.m[2][1] = up.z, v.m[2][2] = forward.z;
v.m[3][0] = -dot(eye, right), v.m[3][1] = -dot(eye, up), v.m[3][2] = -dot(eye, forward), v.m[3][3] = 1;
return v;
}
inline M4 projection_matrix(float fovY, float aspect, float zNear, float zFar)
{
const float ys = 1.0f / std::tan(fovY * 0.5f);
M4 p = {};
p.m[0][0] = ys / aspect;
p.m[1][1] = ys;
p.m[2][2] = zFar / (zFar - zNear);
p.m[2][3] = 1;
p.m[3][2] = -zNear * zFar / (zFar - zNear);
return p;
}
inline void transform(const M4& m, V3 v, float out[4])
{
for (int c = 0; c < 4; ++c)
out[c] = v.x * m.m[0][c] + v.y * m.m[1][c] + v.z * m.m[2][c] + m.m[3][c];
}
// --- the synthetic scene ----------------------------------------------------------
struct Camera
{
V3 eye, target;
};
struct Sphere
{
V3 center;
float radius;
V3 base;
float roughness, metal;
};
inline const Sphere kSpheres[] = {
{{0.0f, 0.2f, 6.0f}, 1.2f, {0.8f, 0.3f, 0.2f}, 0.35f, 0.0f},
{{2.4f, -0.3f, 7.5f}, 0.7f, {0.95f, 0.9f, 0.8f}, 0.15f, 1.0f},
{{-2.2f, -0.4f, 5.0f}, 0.6f, {0.2f, 0.5f, 0.85f}, 0.6f, 0.0f},
};
inline constexpr float kGroundY = -1.0f;
inline constexpr float kWallZ = 14.0f;
// Optional near-mirror pond in the floor (rr_backend_check --water): the reflection is the traced scene, with
// guides as Alan Wake 2 hands them for its water (flat diffuse albedo 0.5, specular albedo ~0.96, roughness ~0.047).
inline bool g_water = false;
inline constexpr float kWaterRoughness = 0.047f;
inline constexpr float kWaterMinX = -4.0f, kWaterMaxX = 4.5f, kWaterMinZ = 0.5f, kWaterMaxZ = 4.8f;
inline const V3 kLight = normalize({0.45f, 0.9f, -0.35f});
struct Hit
{
float t = 1e30f;
V3 position{}, normal{}, base{};
float roughness = 1, metal = 0;
bool valid = false, water = false;
};
inline bool hit_sphere(V3 origin, V3 direction, const Sphere& s, float& t)
{
const V3 oc = origin - s.center;
const float b = dot(oc, direction);
const float c = dot(oc, oc) - s.radius * s.radius;
const float h = b * b - c;
if (h < 0)
return false;
const float root = -b - std::sqrt(h);
if (root <= 1e-3f)
return false;
t = root;
return true;
}
inline Hit trace(V3 origin, V3 direction)
{
Hit hit;
float t;
for (const Sphere& s : kSpheres)
if (hit_sphere(origin, direction, s, t) && t < hit.t)
{
hit.t = t;
hit.position = origin + direction * t;
hit.normal = normalize(hit.position - s.center);
hit.base = s.base;
hit.roughness = s.roughness;
hit.metal = s.metal;
hit.valid = true;
}
if (direction.y < -1e-4f)
{
t = (kGroundY - origin.y) / direction.y;
if (t > 1e-3f && t < hit.t)
{
hit.t = t;
hit.position = origin + direction * t;
hit.normal = {0, 1, 0};
const int checker = (static_cast<int>(std::floor(hit.position.x)) + static_cast<int>(std::floor(hit.position.z))) & 1;
hit.base = checker ? V3{0.75f, 0.75f, 0.72f} : V3{0.22f, 0.24f, 0.25f};
hit.roughness = 0.55f;
hit.metal = 0;
hit.valid = true;
const V3 q = hit.position;
if (g_water && q.x > kWaterMinX && q.x < kWaterMaxX && q.z > kWaterMinZ && q.z < kWaterMaxZ)
{
// small world-space ripples, so the reflection has fine detail a denoiser can lose
hit.normal = normalize({0.035f * std::sin(5.1f * q.x + 1.3f * q.z) + 0.02f * std::sin(11.7f * q.z),
1.0f,
0.035f * std::cos(4.3f * q.z - 0.7f * q.x) + 0.02f * std::cos(13.1f * q.x)});
hit.base = {0.5f, 0.5f, 0.5f};
hit.roughness = kWaterRoughness;
hit.water = true;
}
}
}
if (direction.z > 1e-4f)
{
t = (kWallZ - origin.z) / direction.z;
if (t > 1e-3f && t < hit.t)
{
hit.t = t;
hit.position = origin + direction * t;
hit.normal = {0, 0, -1};
hit.base = {0.55f, 0.5f, 0.45f};
hit.roughness = 0.85f;
hit.metal = 0;
hit.valid = true;
}
}
return hit;
}
inline bool shadowed(V3 position, V3 normal)
{
float t;
const V3 origin = position + normal * 2e-3f;
for (const Sphere& s : kSpheres)
if (hit_sphere(origin, kLight, s, t))
return true;
return false;
}
// AMD's ApproximateBRDF (denoising.md 5.5.2, after Ray Tracing Gems ch. 32).
inline V3 approximate_brdf(V3 f0, float alpha, float nov)
{
nov = std::fabs(nov);
const float x[4] = {1, nov, nov * nov, nov * nov * nov};
const float y[4] = {1, alpha, alpha * alpha, alpha * alpha * alpha};
const float m1[2][2] = {{0.99044f, -1.28514f}, {1.29678f, -0.755907f}};
const float m2[3][3] = {{1.0f, 2.92338f, 59.4188f}, {20.3225f, -27.0302f, 222.592f}, {121.563f, 626.13f, 316.627f}};
const float m3[2][2] = {{0.0365463f, 3.32707f}, {9.0632f, -9.04756f}};
const float m4[3][3] = {{1.0f, 3.59685f, -1.36772f}, {9.04401f, -16.3174f, 9.22949f}, {5.56589f, 19.7886f, -20.2123f}};
// HLSL mul(M, v) with M row-major: (M v)_i = sum_j M[i][j] v_j
const float a0 = m1[0][0] * x[0] + m1[0][1] * x[1], a1 = m1[1][0] * x[0] + m1[1][1] * x[1];
const float xw[3] = {x[0], x[1], x[3]}, yw[3] = {y[0], y[1], y[3]};
float b[3];
for (int i = 0; i < 3; ++i)
b[i] = m2[i][0] * xw[0] + m2[i][1] * xw[1] + m2[i][2] * xw[2];
const float c0 = m3[0][0] * x[0] + m3[0][1] * x[1], c1 = m3[1][0] * x[0] + m3[1][1] * x[1];
const float xz[3] = {x[0], x[2], x[3]};
float d[3];
for (int i = 0; i < 3; ++i)
d[i] = m4[i][0] * xz[0] + m4[i][1] * xz[1] + m4[i][2] * xz[2];
float bias = (a0 * y[0] + a1 * y[1]) / (b[0] * yw[0] + b[1] * yw[1] + b[2] * yw[2]);
const float scale = (c0 * y[0] + c1 * y[1]) / (d[0] * yw[0] + d[1] * yw[1] + d[2] * yw[2]);
bias *= saturate(f0.y * 50);
const float s = std::max(0.0f, scale), bb = std::max(0.0f, bias);
return {f0.x * s + bb, f0.y * s + bb, f0.z * s + bb};
}
inline V3 reflect_dir(V3 direction, V3 normal) { return direction - normal * (2 * dot(direction, normal)); }
inline V3 environment(V3 direction)
{
const float up = saturate(direction.y * 0.5f + 0.5f);
return V3{0.35f, 0.45f, 0.6f} * (0.4f + 1.2f * up);
}
inline uint32_t hash32(uint32_t x)
{
// lowbias32, as AMD's guide recommends for white noise
x ^= x >> 16;
x *= 0x7feb352du;
x ^= x >> 15;
x *= 0x846ca68bu;
x ^= x >> 16;
return x;
}
inline float uniform(uint32_t pixel, uint32_t frame, uint32_t stream)
{
return (hash32(pixel * 0x9e3779b9u ^ hash32(frame * 4u + stream + 0x1234567u)) >> 8) * (1.0f / 16777216.0f);
}
enum Signal
{
kDirectDiffuse,
kDirectSpecular,
kIndirectDiffuse,
kIndirectSpecular,
kSignalCount
};
// FfxApiDenoiserSignalFlags of each signal (DIRECT_DIFFUSE 2, DIRECT_SPECULAR 4, INDIRECT_DIFFUSE 16,
// INDIRECT_SPECULAR 32)
inline const uint32_t kSignalFlags[kSignalCount] = {1u << 1, 1u << 2, 1u << 4, 1u << 5};
inline const char* const kSignalNames[kSignalCount] = {"direct_diffuse", "direct_specular", "indirect_diffuse",
"indirect_specular"};
struct Frame
{
// FSR Ray Regeneration's own inputs (fsr_rr_probe)
std::vector<float> depth; // signed linear (view z)
std::vector<uint16_t> motion; // RGBA16F: prevUV - curUV, prev z - cur z
std::vector<uint32_t> normals; // R10G10B10A2: oct normal, roughness, material 0
std::vector<uint32_t> diffuseAlbedo, specularAlbedo; // RGBA8, sqrt encoded
std::vector<float> clean[kSignalCount]; // RGB
std::vector<uint16_t> noisy[kSignalCount]; // RGBA16F
std::vector<uint8_t> valid;
std::vector<V3> diffuseLinear, specularLinear;
// What a DLSS Ray Reconstruction game hands NGX (rr_backend_check)
std::vector<uint16_t> color; // RGBA16F: the noisy signals remodulated and summed, sky added
std::vector<V3> colorClean; // the same from the noise-free signals: the ground truth
std::vector<float> hardwareDepth; // D32: NDC z of the projection (1 for the sky)
std::vector<uint16_t> motionPixels; // RG16F: (previous - current) position in render pixels
std::vector<uint16_t> normalsRoughness; // RGBA16F: signed world normal, w = linear roughness
std::vector<uint32_t> diffuseAlbedoLinear, specularAlbedoLinear; // RGBA8 UNORM, linear
};
inline uint32_t pack_unorm8(V3 v)
{
auto q = [](float f) { return static_cast<uint32_t>(std::lround(saturate(f) * 255.0f)); };
return q(v.x) | (q(v.y) << 8) | (q(v.z) << 16) | (255u << 24);
}
inline uint32_t pack_normal(V3 n, float roughness)
{
// AMD's NormalToOctahedronUv (denoising.md 5.3.1)
const float l1 = std::fabs(n.x) + std::fabs(n.y) + std::fabs(n.z);
float x = n.x / l1, y = n.y / l1;
if (n.z < 0)
{
const float ox = (1.0f - std::fabs(y)) * (x >= 0 ? 1.0f : -1.0f);
const float oy = (1.0f - std::fabs(x)) * (y >= 0 ? 1.0f : -1.0f);
x = ox, y = oy;
}
auto q10 = [](float f) { return static_cast<uint32_t>(std::lround(saturate(f) * 1023.0f)); };
return q10(x * 0.5f + 0.5f) | (q10(y * 0.5f + 0.5f) << 10) | (q10(roughness) << 20) | (0u << 30);
}
// Renders one frame. (jitterX, jitterY): the sample offset from each pixel's centre in pixels (DLSS's
// Jitter.Offset is its negation). Motion is that of the sampled surface point.
inline void render(Frame& frame, uint32_t width, uint32_t height, const Camera& camera, const Camera& previous,
const M4& projection, uint32_t frameIndex, float jitterX = 0.0f, float jitterY = 0.0f)
{
const size_t count = static_cast<size_t>(width) * height;
frame.depth.assign(count, 0);
frame.motion.assign(count * 4, 0);
frame.normals.assign(count, 0);
frame.diffuseAlbedo.assign(count, 0);
frame.specularAlbedo.assign(count, 0);
frame.valid.assign(count, 0);
frame.diffuseLinear.assign(count, V3{});
frame.specularLinear.assign(count, V3{});
frame.color.assign(count * 4, 0);
frame.colorClean.assign(count, V3{});
frame.hardwareDepth.assign(count, 1.0f);
frame.motionPixels.assign(count * 2, 0);
frame.normalsRoughness.assign(count * 4, 0);
frame.diffuseAlbedoLinear.assign(count, 0);
frame.specularAlbedoLinear.assign(count, 0);
for (int s = 0; s < kSignalCount; ++s)
{
frame.clean[s].assign(count * 3, 0);
frame.noisy[s].assign(count * 4, 0);
}
const M4 view = view_matrix(camera.eye, camera.target);
const M4 previousView = view_matrix(previous.eye, previous.target);
const V3 forward = normalize(camera.target - camera.eye);
const V3 right = normalize(cross({0, 1, 0}, forward));
const V3 up = cross(forward, right);
const float tanX = 1.0f / projection.m[0][0], tanY = 1.0f / projection.m[1][1];
for (uint32_t py = 0; py < height; ++py)
for (uint32_t px = 0; px < width; ++px)
{
const size_t i = static_cast<size_t>(py) * width + px;
const float u = (px + 0.5f + jitterX) / width, v = (py + 0.5f + jitterY) / height;
const float ndcX = u * 2 - 1, ndcY = 1 - v * 2;
const V3 direction = normalize(forward + right * (ndcX * tanX) + up * (ndcY * tanY));
const Hit hit = trace(camera.eye, direction);
const uint32_t pixelId = static_cast<uint32_t>(i);
frame.color[i * 4 + 3] = to_half(1.0f);
frame.normalsRoughness[i * 4 + 2] = to_half(-1.0f);
frame.normalsRoughness[i * 4 + 3] = to_half(1.0f);
if (!hit.valid)
{
// Sky: outside linearDepthBounds, so the runtime passes it through.
frame.depth[i] = 5000.0f;
frame.normals[i] = pack_normal({0, 0, -1}, 1);
const V3 sky = environment(direction);
frame.colorClean[i] = sky;
frame.color[i * 4 + 0] = to_half(sky.x);
frame.color[i * 4 + 1] = to_half(sky.y);
frame.color[i * 4 + 2] = to_half(sky.z);
continue;
}
frame.valid[i] = 1;
float clip[4];
transform(view, hit.position, clip);
frame.depth[i] = clip[2];
float projected[4];
transform(projection, {clip[0], clip[1], clip[2]}, projected);
frame.hardwareDepth[i] = projected[2] / projected[3];
float prevView[4], prevClip[4];
transform(previousView, hit.position, prevView);
const M4& p = projection;
const V3 pv = {prevView[0], prevView[1], prevView[2]};
transform(p, pv, prevClip);
const float prevU = (prevClip[0] / prevClip[3]) * 0.5f + 0.5f;
const float prevV = 0.5f - (prevClip[1] / prevClip[3]) * 0.5f;
frame.motion[i * 4 + 0] = to_half(prevU - u);
frame.motion[i * 4 + 1] = to_half(prevV - v);
frame.motion[i * 4 + 2] = to_half(prevView[2] - clip[2]);
frame.motion[i * 4 + 3] = to_half(0);
frame.motionPixels[i * 2 + 0] = to_half((prevU - u) * width);
frame.motionPixels[i * 2 + 1] = to_half((prevV - v) * height);
const V3 diffuse = hit.base * (1.0f - hit.metal);
const V3 f0 = V3{0.04f, 0.04f, 0.04f} * (1.0f - hit.metal) + hit.base * hit.metal;
const V3 toEye = normalize(camera.eye - hit.position);
const float nov = std::max(dot(hit.normal, toEye), 1e-3f);
const V3 specular = hit.water ? V3{0.96f, 0.96f, 0.96f} : approximate_brdf(f0, hit.roughness * hit.roughness, nov);
frame.diffuseLinear[i] = diffuse;
frame.specularLinear[i] = specular;
frame.diffuseAlbedo[i] = pack_unorm8({std::sqrt(diffuse.x), std::sqrt(diffuse.y), std::sqrt(diffuse.z)});
frame.specularAlbedo[i] = pack_unorm8({std::sqrt(specular.x), std::sqrt(specular.y), std::sqrt(specular.z)});
frame.diffuseAlbedoLinear[i] = pack_unorm8(diffuse);
frame.specularAlbedoLinear[i] = pack_unorm8(specular);
frame.normals[i] = pack_normal(hit.normal, hit.roughness);
frame.normalsRoughness[i * 4 + 0] = to_half(hit.normal.x);
frame.normalsRoughness[i * 4 + 1] = to_half(hit.normal.y);
frame.normalsRoughness[i * 4 + 2] = to_half(hit.normal.z);
frame.normalsRoughness[i * 4 + 3] = to_half(hit.roughness);
// Demodulated (albedo-free) clean signals, smooth in world space.
const float visibility = shadowed(hit.position, hit.normal) ? 0.0f : 1.0f;
const float ndl = std::max(dot(hit.normal, kLight), 0.0f);
const V3 reflected = direction - hit.normal * (2 * dot(direction, hit.normal));
const float rdl = std::max(dot(reflected, kLight), 0.0f);
const float glossExponent = 2.0f / (hit.roughness * hit.roughness * hit.roughness * hit.roughness + 0.02f);
V3 clean[kSignalCount];
clean[kDirectDiffuse] = V3{1.0f, 0.95f, 0.85f} * (2.5f * ndl * visibility);
clean[kDirectSpecular] = V3{1.0f, 0.95f, 0.85f} * std::min(8.0f, 3.0f * std::pow(rdl, glossExponent) * visibility * (ndl > 0));
const float wobble = 0.12f * std::sin(1.7f * hit.position.x) * std::sin(1.3f * hit.position.z);
clean[kIndirectDiffuse] = environment(hit.normal) * (0.6f + wobble);
clean[kIndirectSpecular] = environment(reflected) * (0.8f + 0.2f * (1 - hit.roughness));
if (hit.water)
{
// The pond's light is its reflection: the traced scene, lit noise-free. Its diffuse lobe is
// nearly dark despite the 0.5 guide albedo.
clean[kDirectDiffuse] = clean[kDirectDiffuse] * 0.01f;
clean[kIndirectDiffuse] = clean[kIndirectDiffuse] * 0.01f;
const Hit bounce = trace(hit.position + hit.normal * 2e-3f, reflected);
if (bounce.valid && !bounce.water)
{
const float bounceVisibility = shadowed(bounce.position, bounce.normal) ? 0.0f : 1.0f;
const float bounceNdl = std::max(dot(bounce.normal, kLight), 0.0f);
const V3 light = V3{1.0f, 0.95f, 0.85f} * (2.5f * bounceNdl * bounceVisibility) +
environment(bounce.normal) * 0.6f;
clean[kIndirectSpecular] = mul(bounce.base * (1.0f - bounce.metal), light) +
environment(reflect_dir(reflected, bounce.normal)) * (0.04f + 0.9f * bounce.metal);
}
else
clean[kIndirectSpecular] = environment(reflected);
}
V3 combined{}, combinedClean{};
for (int s = 0; s < kSignalCount; ++s)
{
// One-sample-per-pixel white noise: a mean-one exponential weight shared by RGB,
// like a single path sample's throughput.
float sample = -std::log(std::max(1e-6f, 1.0f - uniform(pixelId, frameIndex, s)));
if (hit.water && s == kIndirectSpecular)
sample = 1.0f + 0.35f * (sample - 1.0f); // a mirror's reflection ray is far less noisy
frame.clean[s][i * 3 + 0] = clean[s].x;
frame.clean[s][i * 3 + 1] = clean[s].y;
frame.clean[s][i * 3 + 2] = clean[s].z;
frame.noisy[s][i * 4 + 0] = to_half(clean[s].x * sample);
frame.noisy[s][i * 4 + 1] = to_half(clean[s].y * sample);
frame.noisy[s][i * 4 + 2] = to_half(clean[s].z * sample);
// A: ray hit distance for the indirect signals, any non-negative value for direct.
float alpha = 0.0f;
if (s == kIndirectDiffuse)
alpha = 0.5f + 3.0f * uniform(pixelId, frameIndex, 10 + s);
else if (s == kIndirectSpecular)
{
const Hit bounce = trace(hit.position + hit.normal * 2e-3f, reflected);
alpha = bounce.valid ? std::min(bounce.t, 100.0f) : 100.0f;
}
frame.noisy[s][i * 4 + 3] = to_half(alpha);
const V3 albedo = s == kDirectDiffuse || s == kIndirectDiffuse ? diffuse : specular;
combined = combined + mul(clean[s] * sample, albedo);
combinedClean = combinedClean + mul(clean[s], albedo);
}
frame.colorClean[i] = combinedClean;
frame.color[i * 4 + 0] = to_half(combined.x);
frame.color[i * 4 + 1] = to_half(combined.y);
frame.color[i * 4 + 2] = to_half(combined.z);
}
}
} // namespace fsr_scene
tools/rr_backend_check.cpp
+733
View File
@@ -0,0 +1,733 @@
// Drives d4r_nvngx.dll the way a DLSS Ray Reconstruction game does, with the synthetic ray-traced
// scene of fsr_rr_scene.h, and measures the result against ground truth: the scene's noise-free image
// rendered directly at the output resolution. It exercises whichever Ray Reconstruction backend the
// shim is configured with ([RayReconstruction] Backend / D4R_RR_BACKEND), or Super Sampling alone on
// the same noisy colour (--feature 1) as the no-denoiser baseline.
//
// Per frame the game-side inputs are: the noisy colour (each signal remodulated by its albedo, summed),
// linear or hardware depth, render-resolution motion vectors in pixels (previous - current), world
// normals with the roughness in w (DLSS.Roughness.Mode 1), linear diffuse and specular albedos, the
// unjittered WorldToView/ViewToClip matrices, and Halton(2,3) jitter. The camera holds still, then
// pans, then cuts (Reset) to another view.
//
// It checks: capability and requirement answers, feature creation, every evaluation's result, output
// written at exactly the requested size, finite values, error against ground truth before and after
// the cut, release and shutdown. Timing: CPU time of each EvaluateFeature call and of submit + wait.
//
// usage: rr_backend_check.exe SHIM_DLL OUT_DIR [--render WxH] [--output WxH] [--frames N] [--feature 13|1]
// [--hw-depth] [--separate-roughness] [--specular-hit]
#define WIDL_EXPLICIT_AGGREGATE_RETURNS
#include <windows.h>
#include <d3d12.h>
#include <dxgi1_4.h>
#include <algorithm>
#include <chrono>
#include <cmath>
#include <cstdarg>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
#include <vector>
#include "fsr_rr_scene.h"
using namespace fsr_scene;
using NgxResult = unsigned int;
constexpr NgxResult NGX_SUCCESS = 1;
struct NgxHandle
{
unsigned int Id;
};
extern "C"
{
void d4r_ngx_set_float(void* parameters, const char* name, float value);
void d4r_ngx_set_uint(void* parameters, const char* name, unsigned int value);
void d4r_ngx_set_int(void* parameters, const char* name, int value);
void d4r_ngx_set_d3d12_resource(void* parameters, const char* name, ID3D12Resource* value);
void d4r_ngx_set_void(void* parameters, const char* name, void* value);
NgxResult d4r_ngx_get_int(void* parameters, const char* name, int* value);
NgxResult d4r_ngx_get_void(void* parameters, const char* name, void** value);
}
using PFN_Init_Ext = NgxResult (*)(unsigned long long, const wchar_t*, ID3D12Device*, unsigned int, const void*);
using PFN_Parameters = NgxResult (*)(void**);
using PFN_CreateFeature = NgxResult (*)(ID3D12GraphicsCommandList*, unsigned int, void*, NgxHandle**);
using PFN_Evaluate = NgxResult (*)(ID3D12GraphicsCommandList*, const NgxHandle*, void*, void*);
using PFN_Release = NgxResult (*)(NgxHandle*);
using PFN_Shutdown = NgxResult (*)();
struct NgxApplicationIdentifier
{
unsigned int type;
union
{
unsigned long long applicationId;
struct
{
const char* id;
int engineType;
const char* engineVersion;
} project;
} value;
};
struct NgxFeatureDiscovery
{
unsigned int sdkVersion;
unsigned int feature;
NgxApplicationIdentifier identifier;
const wchar_t* dataPath;
const void* featureInfo;
};
struct NgxFeatureRequirement
{
unsigned int supported;
unsigned int minArchitecture;
char minOsVersion[255];
};
using PFN_Requirements = NgxResult (*)(void*, const NgxFeatureDiscovery*, NgxFeatureRequirement*);
namespace
{
ID3D12Device* g_device;
ID3D12CommandQueue* g_queue;
ID3D12CommandAllocator* g_allocator;
ID3D12GraphicsCommandList* g_list;
ID3D12Fence* g_fence;
HANDLE g_event;
UINT64 g_fenceValue;
FILE* g_json;
bool g_jsonFirst = true;
std::string format(const char* pattern, ...)
{
char buffer[4096];
va_list args;
va_start(args, pattern);
std::vsnprintf(buffer, sizeof(buffer), pattern, args);
va_end(args);
return buffer;
}
void json_field(const char* key, const std::string& value)
{
if (g_json)
std::fprintf(g_json, "%s\n \"%s\": %s", g_jsonFirst ? "" : ",", key, value.c_str());
g_jsonFirst = false;
}
bool g_failed = false;
void check(const char* name, bool ok, const std::string& detail)
{
std::printf("check=%s result=%s %s\n", name, ok ? "ok" : "FAIL", detail.c_str());
std::fflush(stdout);
std::string escaped;
for (char c : detail)
if (c == '"' || c == '\\')
escaped += '\\', escaped += c;
else if (static_cast<unsigned char>(c) >= 0x20)
escaped += c;
json_field((std::string("check_") + name).c_str(),
format("{\"ok\": %s, \"detail\": \"%s\"}", ok ? "true" : "false", escaped.c_str()));
g_failed |= !ok;
}
// Process VRAM from the DRM fdinfo of this process (Wine maps Z: to /), as the D3D12 harness does.
unsigned long long process_vram_kib()
{
std::vector<unsigned long long> clients;
unsigned long long total = 0;
for (int fd = 0; fd < 4096; ++fd)
{
char path[64];
std::snprintf(path, sizeof(path), "Z:\\proc\\self\\fdinfo\\%d", fd);
FILE* file = std::fopen(path, "r");
if (file == nullptr)
continue;
unsigned long long client = 0, vram = 0;
bool drm = false;
char line[256];
while (std::fgets(line, sizeof(line), file) != nullptr)
if (std::sscanf(line, "drm-client-id: %llu", &client) == 1)
drm = true;
else
std::sscanf(line, "drm-memory-vram: %llu", &vram);
std::fclose(file);
if (drm && std::find(clients.begin(), clients.end(), client) == clients.end())
{
clients.push_back(client);
total += vram;
}
}
return total;
}
ID3D12Resource* create_texture(UINT width, UINT height, DXGI_FORMAT format, D3D12_RESOURCE_FLAGS flags,
D3D12_RESOURCE_STATES state)
{
D3D12_RESOURCE_DESC desc = {};
desc.Dimension = D3D12_RESOURCE_DIMENSION_TEXTURE2D;
desc.Width = width;
desc.Height = height;
desc.DepthOrArraySize = 1;
desc.MipLevels = 1;
desc.Format = format;
desc.SampleDesc.Count = 1;
desc.Flags = flags;
D3D12_HEAP_PROPERTIES heap = {};
heap.Type = D3D12_HEAP_TYPE_DEFAULT;
ID3D12Resource* resource = nullptr;
if (FAILED(g_device->CreateCommittedResource(&heap, D3D12_HEAP_FLAG_NONE, &desc, state, nullptr,
__uuidof(ID3D12Resource), reinterpret_cast<void**>(&resource))))
return nullptr;
return resource;
}
ID3D12Resource* create_buffer(UINT64 bytes, D3D12_HEAP_TYPE type, uint8_t** mapped)
{
D3D12_RESOURCE_DESC desc = {};
desc.Dimension = D3D12_RESOURCE_DIMENSION_BUFFER;
desc.Width = bytes;
desc.Height = 1;
desc.DepthOrArraySize = 1;
desc.MipLevels = 1;
desc.SampleDesc.Count = 1;
desc.Layout = D3D12_TEXTURE_LAYOUT_ROW_MAJOR;
D3D12_HEAP_PROPERTIES heap = {};
heap.Type = type;
ID3D12Resource* buffer = nullptr;
const D3D12_RESOURCE_STATES state =
type == D3D12_HEAP_TYPE_UPLOAD ? D3D12_RESOURCE_STATE_GENERIC_READ : D3D12_RESOURCE_STATE_COPY_DEST;
if (FAILED(g_device->CreateCommittedResource(&heap, D3D12_HEAP_FLAG_NONE, &desc, state, nullptr,
__uuidof(ID3D12Resource), reinterpret_cast<void**>(&buffer))) ||
FAILED(buffer->Map(0, nullptr, reinterpret_cast<void**>(mapped))))
return nullptr;
return buffer;
}
void transition(ID3D12Resource* resource, D3D12_RESOURCE_STATES before, D3D12_RESOURCE_STATES after)
{
if (before == after)
return;
D3D12_RESOURCE_BARRIER barrier = {};
barrier.Type = D3D12_RESOURCE_BARRIER_TYPE_TRANSITION;
barrier.Transition.pResource = resource;
barrier.Transition.Subresource = D3D12_RESOURCE_BARRIER_ALL_SUBRESOURCES;
barrier.Transition.StateBefore = before;
barrier.Transition.StateAfter = after;
g_list->ResourceBarrier(1, &barrier);
}
bool submit_and_wait()
{
if (FAILED(g_list->Close()))
return false;
ID3D12CommandList* lists[] = {g_list};
g_queue->ExecuteCommandLists(1, lists);
g_queue->Signal(g_fence, ++g_fenceValue);
g_fence->SetEventOnCompletion(g_fenceValue, g_event);
const bool done = WaitForSingleObject(g_event, 30000) == WAIT_OBJECT_0;
g_allocator->Reset();
g_list->Reset(g_allocator, nullptr);
return done && SUCCEEDED(g_device->GetDeviceRemovedReason());
}
// Uploads tightly packed rows into a texture (left in `state`).
struct Uploader
{
ID3D12Resource* staging = nullptr;
uint8_t* mapped = nullptr;
UINT64 capacity = 0, offset = 0;
void upload(ID3D12Resource* texture, UINT width, UINT height, DXGI_FORMAT format, const void* data,
size_t texelBytes, D3D12_RESOURCE_STATES state)
{
D3D12_RESOURCE_DESC desc = {};
desc.Dimension = D3D12_RESOURCE_DIMENSION_TEXTURE2D;
desc.Width = width;
desc.Height = height;
desc.DepthOrArraySize = 1;
desc.MipLevels = 1;
desc.Format = format;
desc.SampleDesc.Count = 1;
D3D12_PLACED_SUBRESOURCE_FOOTPRINT footprint = {};
UINT rows = 0;
UINT64 rowBytes = 0, total = 0;
g_device->GetCopyableFootprints(&desc, 0, 1, 0, &footprint, &rows, &rowBytes, &total);
offset = (offset + 511) & ~UINT64(511);
footprint.Offset = offset;
for (UINT row = 0; row < rows; ++row)
std::memcpy(mapped + offset + row * footprint.Footprint.RowPitch,
static_cast<const uint8_t*>(data) + row * texelBytes * width, texelBytes * width);
offset += total;
transition(texture, state, D3D12_RESOURCE_STATE_COPY_DEST);
D3D12_TEXTURE_COPY_LOCATION dst = {};
dst.pResource = texture;
dst.Type = D3D12_TEXTURE_COPY_TYPE_SUBRESOURCE_INDEX;
D3D12_TEXTURE_COPY_LOCATION src = {};
src.pResource = staging;
src.Type = D3D12_TEXTURE_COPY_TYPE_PLACED_FOOTPRINT;
src.PlacedFootprint = footprint;
g_list->CopyTextureRegion(&dst, 0, 0, 0, &src, nullptr);
transition(texture, D3D12_RESOURCE_STATE_COPY_DEST, state);
}
};
float halton(uint32_t index, uint32_t base)
{
float result = 0, fraction = 1;
while (index > 0)
{
fraction /= base;
result += fraction * (index % base);
index /= base;
}
return result;
}
void write_ppm(const std::string& path, uint32_t width, uint32_t height, const std::vector<V3>& radiance)
{
FILE* file = std::fopen(path.c_str(), "wb");
if (file == nullptr)
return;
std::fprintf(file, "P6\n%u %u\n255\n", width, height);
std::vector<uint8_t> row(width * 3);
for (uint32_t y = 0; y < height; ++y)
{
for (uint32_t x = 0; x < width; ++x)
{
const V3 c = radiance[static_cast<size_t>(y) * width + x];
const float rgb[3] = {c.x, c.y, c.z};
for (int k = 0; k < 3; ++k)
{
const float v = std::max(0.0f, rgb[k]);
row[x * 3 + k] = static_cast<uint8_t>(std::lround(saturate(std::pow(v / (1 + v), 1 / 2.2f)) * 255));
}
}
std::fwrite(row.data(), 1, row.size(), file);
}
std::fclose(file);
}
} // namespace
int main(int argc, char** argv)
{
if (argc < 3)
{
std::printf("usage: rr_backend_check.exe SHIM_DLL OUT_DIR [--render WxH] [--output WxH] [--frames N] "
"[--feature 13|1] [--hw-depth] [--separate-roughness] [--specular-hit] [--separate-eval-params] [--water] [--measure-all]\n");
return 2;
}
const std::string outDir = argv[2];
UINT rw = 960, rh = 540, ow = 1920, oh = 1080, frames = 48, featureId = 13;
bool hwDepth = false, separateRoughness = false, specularHit = false, separateEvalParams = false, measureAll = false;
for (int i = 3; i < argc; ++i)
{
const std::string arg = argv[i];
if (arg == "--render" && i + 1 < argc)
std::sscanf(argv[++i], "%ux%u", &rw, &rh);
else if (arg == "--output" && i + 1 < argc)
std::sscanf(argv[++i], "%ux%u", &ow, &oh);
else if (arg == "--frames" && i + 1 < argc)
frames = static_cast<UINT>(std::atoi(argv[++i]));
else if (arg == "--feature" && i + 1 < argc)
featureId = static_cast<UINT>(std::atoi(argv[++i]));
else if (arg == "--hw-depth")
hwDepth = true;
else if (arg == "--separate-roughness")
separateRoughness = true;
else if (arg == "--specular-hit")
specularHit = true;
else if (arg == "--separate-eval-params")
separateEvalParams = true;
else if (arg == "--water")
fsr_scene::g_water = true;
else if (arg == "--measure-all")
measureAll = true;
}
const bool rr = featureId == 13;
CreateDirectoryA(outDir.c_str(), nullptr);
g_json = std::fopen((outDir + "\\check.json").c_str(), "w");
if (g_json)
std::fputs("{", g_json);
json_field("feature", format("%u", featureId));
json_field("render_size", format("[%u, %u]", rw, rh));
json_field("output_size", format("[%u, %u]", ow, oh));
json_field("options", format("{\"hw_depth\": %s, \"separate_roughness\": %s, \"specular_hit\": %s, \"separate_eval_params\": %s}",
hwDepth ? "true" : "false", separateRoughness ? "true" : "false",
specularHit ? "true" : "false", separateEvalParams ? "true" : "false"));
auto finish = [&](int code) {
if (g_json)
{
std::fputs("\n}\n", g_json);
std::fclose(g_json);
}
std::printf("overall=%s\n", code == 0 ? "PASS" : "FAIL");
return code;
};
if (FAILED(D3D12CreateDevice(nullptr, D3D_FEATURE_LEVEL_12_0, __uuidof(ID3D12Device), reinterpret_cast<void**>(&g_device))))
{
check("device", false, "D3D12CreateDevice failed");
return finish(1);
}
D3D12_COMMAND_QUEUE_DESC queueDesc = {};
queueDesc.Type = D3D12_COMMAND_LIST_TYPE_DIRECT;
g_device->CreateCommandQueue(&queueDesc, __uuidof(ID3D12CommandQueue), reinterpret_cast<void**>(&g_queue));
g_device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_DIRECT, __uuidof(ID3D12CommandAllocator),
reinterpret_cast<void**>(&g_allocator));
g_device->CreateCommandList(0, D3D12_COMMAND_LIST_TYPE_DIRECT, g_allocator, nullptr,
__uuidof(ID3D12GraphicsCommandList), reinterpret_cast<void**>(&g_list));
g_device->CreateFence(0, D3D12_FENCE_FLAG_NONE, __uuidof(ID3D12Fence), reinterpret_cast<void**>(&g_fence));
g_event = CreateEventW(nullptr, FALSE, FALSE, nullptr);
const unsigned long long vramStart = process_vram_kib();
HMODULE shim = LoadLibraryA(argv[1]);
if (shim == nullptr)
{
check("load", false, format("LoadLibrary(%s) failed: %lu", argv[1], GetLastError()));
return finish(1);
}
auto init = reinterpret_cast<PFN_Init_Ext>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_Init_Ext")));
auto capability = reinterpret_cast<PFN_Parameters>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_GetCapabilityParameters")));
auto allocate = reinterpret_cast<PFN_Parameters>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_AllocateParameters")));
auto createFeature = reinterpret_cast<PFN_CreateFeature>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_CreateFeature")));
auto evaluate = reinterpret_cast<PFN_Evaluate>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_EvaluateFeature")));
auto release = reinterpret_cast<PFN_Release>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_ReleaseFeature")));
auto shutdown = reinterpret_cast<PFN_Shutdown>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_Shutdown")));
auto requirements = reinterpret_cast<PFN_Requirements>(reinterpret_cast<void*>(GetProcAddress(shim, "NVSDK_NGX_D3D12_GetFeatureRequirements")));
if (!init || !capability || !allocate || !createFeature || !evaluate || !release || !shutdown || !requirements)
{
check("load", false, "missing NGX exports");
return finish(1);
}
// Discovery before Init, as games do.
wchar_t dataPath[MAX_PATH];
GetTempPathW(MAX_PATH, dataPath);
NgxFeatureDiscovery discovery = {};
discovery.sdkVersion = 0x15;
discovery.feature = featureId;
discovery.identifier.value.applicationId = 241534723ULL;
discovery.dataPath = dataPath;
NgxFeatureRequirement requirement = {};
IDXGIFactory4* factory = nullptr;
IDXGIAdapter* adapter = nullptr;
LUID luid;
g_device->GetAdapterLuid(&luid);
CreateDXGIFactory1(__uuidof(IDXGIFactory4), reinterpret_cast<void**>(&factory));
factory->EnumAdapterByLuid(luid, __uuidof(IDXGIAdapter), reinterpret_cast<void**>(&adapter));
const NgxResult discovered = requirements(adapter, &discovery, &requirement);
check("requirements_before_init", discovered == NGX_SUCCESS && requirement.supported == 0,
format("result 0x%08x, FeatureSupported %u", discovered, requirement.supported));
adapter->Release();
factory->Release();
const auto initStart = std::chrono::steady_clock::now();
const NgxResult initialized = init(241534723ULL, dataPath, g_device, 0x15, nullptr);
const double initMs = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - initStart).count();
check("init", initialized == NGX_SUCCESS, format("0x%08x in %.0f ms", initialized, initMs));
if (initialized != NGX_SUCCESS)
return finish(1);
void* capabilities = nullptr;
capability(&capabilities);
int available = -1;
void* optimal = nullptr;
const char* key = rr ? "SuperSamplingDenoising.Available" : "SuperSampling.Available";
d4r_ngx_get_int(capabilities, key, &available);
d4r_ngx_get_void(capabilities, rr ? "DLSSDOptimalSettingsCallback" : "DLSSOptimalSettingsCallback", &optimal);
check("capability", available == 1, format("%s=%d, optimal-settings callback %s", key, available, optimal ? "set" : "absent"));
// Inputs at render size, output at output size.
const auto srv = D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE;
ID3D12Resource* color = create_texture(rw, rh, DXGI_FORMAT_R16G16B16A16_FLOAT, D3D12_RESOURCE_FLAG_NONE, srv);
ID3D12Resource* depth = create_texture(rw, rh, DXGI_FORMAT_R32_FLOAT, D3D12_RESOURCE_FLAG_NONE, srv);
ID3D12Resource* motion = create_texture(rw, rh, DXGI_FORMAT_R16G16_FLOAT, D3D12_RESOURCE_FLAG_NONE, srv);
ID3D12Resource* normals = create_texture(rw, rh, DXGI_FORMAT_R16G16B16A16_FLOAT, D3D12_RESOURCE_FLAG_NONE, srv);
ID3D12Resource* roughness = create_texture(rw, rh, DXGI_FORMAT_R16_FLOAT, D3D12_RESOURCE_FLAG_NONE, srv);
ID3D12Resource* diffuse = create_texture(rw, rh, DXGI_FORMAT_R8G8B8A8_UNORM, D3D12_RESOURCE_FLAG_NONE, srv);
ID3D12Resource* specular = create_texture(rw, rh, DXGI_FORMAT_R8G8B8A8_UNORM, D3D12_RESOURCE_FLAG_NONE, srv);
ID3D12Resource* hitDistance = create_texture(rw, rh, DXGI_FORMAT_R16_FLOAT, D3D12_RESOURCE_FLAG_NONE, srv);
// The output starts out holding a sentinel, so an unwritten texel is detectable.
ID3D12Resource* output = create_texture(ow, oh, DXGI_FORMAT_R16G16B16A16_FLOAT,
D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS, D3D12_RESOURCE_STATE_UNORDERED_ACCESS);
Uploader uploader;
uploader.capacity = 64ull << 20;
uploader.staging = create_buffer(uploader.capacity, D3D12_HEAP_TYPE_UPLOAD, &uploader.mapped);
D3D12_RESOURCE_DESC outputDesc = {};
outputDesc.Dimension = D3D12_RESOURCE_DIMENSION_TEXTURE2D;
outputDesc.Width = ow;
outputDesc.Height = oh;
outputDesc.DepthOrArraySize = 1;
outputDesc.MipLevels = 1;
outputDesc.Format = DXGI_FORMAT_R16G16B16A16_FLOAT;
outputDesc.SampleDesc.Count = 1;
D3D12_PLACED_SUBRESOURCE_FOOTPRINT outputFootprint = {};
UINT outputRows = 0;
UINT64 outputRowBytes = 0, outputTotal = 0;
g_device->GetCopyableFootprints(&outputDesc, 0, 1, 0, &outputFootprint, &outputRows, &outputRowBytes, &outputTotal);
uint8_t* readMapped = nullptr;
ID3D12Resource* readback = create_buffer(outputTotal, D3D12_HEAP_TYPE_READBACK, &readMapped);
if (!color || !depth || !motion || !normals || !roughness || !diffuse || !specular || !hitDistance || !output ||
!uploader.staging || !readback)
{
check("resources", false, "texture creation");
return finish(1);
}
void* parameters = nullptr;
allocate(&parameters);
d4r_ngx_set_uint(parameters, "Width", rw);
d4r_ngx_set_uint(parameters, "Height", rh);
d4r_ngx_set_uint(parameters, "OutWidth", ow);
d4r_ngx_set_uint(parameters, "OutHeight", oh);
d4r_ngx_set_int(parameters, "PerfQualityValue", 2);
d4r_ngx_set_int(parameters, "DLSS.Feature.Create.Flags", 1 | 2); // HDR | MVLowRes
if (rr)
{
d4r_ngx_set_int(parameters, "DLSS.Denoise.Mode", 1);
d4r_ngx_set_int(parameters, "DLSS.Roughness.Mode", separateRoughness ? 0 : 1);
d4r_ngx_set_int(parameters, "DLSS.Use.HW.Depth", hwDepth ? 1 : 0);
}
NgxHandle* feature = nullptr;
const auto createStart = std::chrono::steady_clock::now();
const NgxResult created = createFeature(g_list, featureId, parameters, &feature);
const double createMs = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - createStart).count();
check("create", created == NGX_SUCCESS && feature != nullptr, format("0x%08x in %.0f ms", created, createMs));
json_field("create_ms", format("%.1f", createMs));
if (created != NGX_SUCCESS)
return finish(1);
// Keep the depth/roughness modes only on the creation object, as callers are allowed to do.
// Frame resources and camera settings below go onto a fresh object without those scalars.
if (separateEvalParams)
allocate(&parameters);
const M4 projection = projection_matrix(1.0f, static_cast<float>(rw) / rh, 0.1f, 1000.0f);
const uint32_t moveStart = frames / 3, cutFrame = frames * 2 / 3;
auto camera_at = [&](uint32_t f) -> Camera {
if (f >= cutFrame)
return {{3.5f, 1.6f, 1.5f}, {-0.5f, -0.6f, 7.5f}};
const float t = f >= moveStart ? static_cast<float>(f - moveStart) : 0.0f;
return {{-0.6f + 0.04f * t, 0.6f, -1.5f}, {0.0f + 0.02f * t, -0.1f, 6.5f}};
};
Frame scene, truth;
Camera previous = camera_at(0);
std::vector<double> callUs, totalUs;
std::string log = "[";
bool allEvaluated = true, sized = true, finite = true;
double errorBeforeCut = -1, errorFinal = -1, errorAtCut = -1;
std::vector<uint16_t> hit(static_cast<size_t>(rw) * rh), rough(static_cast<size_t>(rw) * rh);
std::vector<float> view(16), clip(16);
for (uint32_t f = 0; f < frames; ++f)
{
const Camera camera = camera_at(f);
const bool reset = f == 0 || f == cutFrame;
const uint32_t phase = f % 16 + 1;
const float jx = halton(phase, 2) - 0.5f, jy = halton(phase, 3) - 0.5f;
render(scene, rw, rh, camera, reset ? camera : previous, projection, f, jx, jy);
uploader.offset = 0;
uploader.upload(color, rw, rh, DXGI_FORMAT_R16G16B16A16_FLOAT, scene.color.data(), 8, srv);
if (hwDepth)
uploader.upload(depth, rw, rh, DXGI_FORMAT_R32_FLOAT, scene.hardwareDepth.data(), 4, srv);
else
{
std::vector<float> linear(scene.depth);
for (size_t i = 0; i < linear.size(); ++i)
linear[i] = scene.valid[i] ? linear[i] : 1.0e7f; // the sky: beyond any geometry
uploader.upload(depth, rw, rh, DXGI_FORMAT_R32_FLOAT, linear.data(), 4, srv);
}
uploader.upload(motion, rw, rh, DXGI_FORMAT_R16G16_FLOAT, scene.motionPixels.data(), 4, srv);
if (rr)
{
uploader.upload(normals, rw, rh, DXGI_FORMAT_R16G16B16A16_FLOAT, scene.normalsRoughness.data(), 8, srv);
uploader.upload(diffuse, rw, rh, DXGI_FORMAT_R8G8B8A8_UNORM, scene.diffuseAlbedoLinear.data(), 4, srv);
uploader.upload(specular, rw, rh, DXGI_FORMAT_R8G8B8A8_UNORM, scene.specularAlbedoLinear.data(), 4, srv);
if (separateRoughness)
{
for (size_t i = 0; i < rough.size(); ++i)
rough[i] = scene.normalsRoughness[i * 4 + 3];
uploader.upload(roughness, rw, rh, DXGI_FORMAT_R16_FLOAT, rough.data(), 2, srv);
}
if (specularHit)
{
for (size_t i = 0; i < hit.size(); ++i)
hit[i] = scene.noisy[kIndirectSpecular][i * 4 + 3];
uploader.upload(hitDistance, rw, rh, DXGI_FORMAT_R16_FLOAT, hit.data(), 2, srv);
}
}
d4r_ngx_set_d3d12_resource(parameters, "Color", color);
d4r_ngx_set_d3d12_resource(parameters, "Depth", depth);
d4r_ngx_set_d3d12_resource(parameters, "MotionVectors", motion);
d4r_ngx_set_d3d12_resource(parameters, "Output", output);
d4r_ngx_set_float(parameters, "Jitter.Offset.X", -jx); // DLSS: minus the sample offset
d4r_ngx_set_float(parameters, "Jitter.Offset.Y", -jy);
d4r_ngx_set_float(parameters, "MV.Scale.X", 1.0f);
d4r_ngx_set_float(parameters, "MV.Scale.Y", 1.0f);
d4r_ngx_set_int(parameters, "Reset", reset ? 1 : 0);
d4r_ngx_set_uint(parameters, "DLSS.Render.Subrect.Dimensions.Width", rw);
d4r_ngx_set_uint(parameters, "DLSS.Render.Subrect.Dimensions.Height", rh);
if (rr)
{
d4r_ngx_set_d3d12_resource(parameters, "GBuffer.Normals", normals);
d4r_ngx_set_d3d12_resource(parameters, "GBuffer.Roughness", separateRoughness ? roughness : nullptr);
d4r_ngx_set_d3d12_resource(parameters, "GBuffer.DiffuseAlbedo", diffuse);
d4r_ngx_set_d3d12_resource(parameters, "GBuffer.SpecularAlbedo", specular);
d4r_ngx_set_d3d12_resource(parameters, "DLSSD.SpecularHitDistance", specularHit ? hitDistance : nullptr);
const M4 viewMatrix = view_matrix(camera.eye, camera.target);
std::memcpy(view.data(), &viewMatrix, 64);
std::memcpy(clip.data(), &projection, 64);
d4r_ngx_set_void(parameters, "WorldToViewMatrix", view.data());
d4r_ngx_set_void(parameters, "ViewToClipMatrix", clip.data());
}
// Mark the output with a sentinel (NaN) the backend has to overwrite everywhere.
{
std::vector<uint16_t> sentinel(static_cast<size_t>(ow) * oh * 4, 0x7e00u);
uploader.upload(output, ow, oh, DXGI_FORMAT_R16G16B16A16_FLOAT, sentinel.data(), 8,
D3D12_RESOURCE_STATE_UNORDERED_ACCESS);
}
const auto start = std::chrono::steady_clock::now();
const NgxResult evaluated = evaluate(g_list, feature, parameters, nullptr);
const auto recorded = std::chrono::steady_clock::now();
transition(output, D3D12_RESOURCE_STATE_UNORDERED_ACCESS, D3D12_RESOURCE_STATE_COPY_SOURCE);
D3D12_TEXTURE_COPY_LOCATION dst = {};
dst.pResource = readback;
dst.Type = D3D12_TEXTURE_COPY_TYPE_PLACED_FOOTPRINT;
dst.PlacedFootprint = outputFootprint;
D3D12_TEXTURE_COPY_LOCATION src = {};
src.pResource = output;
src.Type = D3D12_TEXTURE_COPY_TYPE_SUBRESOURCE_INDEX;
g_list->CopyTextureRegion(&dst, 0, 0, 0, &src, nullptr);
transition(output, D3D12_RESOURCE_STATE_COPY_SOURCE, D3D12_RESOURCE_STATE_UNORDERED_ACCESS);
const bool submitted = submit_and_wait();
const auto end = std::chrono::steady_clock::now();
callUs.push_back(std::chrono::duration<double, std::micro>(recorded - start).count());
totalUs.push_back(std::chrono::duration<double, std::micro>(end - start).count());
if (evaluated != NGX_SUCCESS || !submitted)
{
allEvaluated = false;
std::printf("frame %u: EvaluateFeature 0x%08x, submit %s\n", f, evaluated, submitted ? "ok" : "FAILED");
if (!submitted)
break;
}
// Ground truth (the noise-free scene rendered at the output size from the same camera), on the frames
// that are reported and every 8th.
const bool measured = measureAll || f % 8 == 7 || f == 0 || f + 1 == moveStart || f + 1 == cutFrame || f == cutFrame ||
f + 1 == frames;
if (!measured)
{
previous = camera;
continue;
}
render(truth, ow, oh, camera, camera, projection_matrix(1.0f, static_cast<float>(ow) / oh, 0.1f, 1000.0f), f);
std::vector<V3> result(static_cast<size_t>(ow) * oh);
double error = 0, noisyError = 0, waterError = 0;
size_t unwritten = 0, nonFinite = 0, n = 0, waterN = 0;
for (UINT y = 0; y < oh; ++y)
for (UINT x = 0; x < ow; ++x)
{
const uint16_t* texel = reinterpret_cast<const uint16_t*>(readMapped + outputFootprint.Offset +
y * outputFootprint.Footprint.RowPitch) + x * 4;
const size_t i = static_cast<size_t>(y) * ow + x;
const V3 value{from_half(texel[0]), from_half(texel[1]), from_half(texel[2])};
if (texel[0] == 0x7e00u && texel[1] == 0x7e00u && texel[2] == 0x7e00u)
{
++unwritten;
continue;
}
if (!std::isfinite(value.x) || !std::isfinite(value.y) || !std::isfinite(value.z))
{
++nonFinite;
continue;
}
result[i] = value;
// Compared in a tonemapped space (x / (1 + x)), so highlights do not dominate.
const V3 t = truth.colorClean[i];
const float tv[3] = {t.x, t.y, t.z}, ov[3] = {value.x, value.y, value.z};
for (int c = 0; c < 3; ++c)
{
const double d = ov[c] / (1 + std::max(0.0f, ov[c])) - tv[c] / (1 + tv[c]);
error += d * d;
++n;
if (from_half(truth.normalsRoughness[i * 4 + 3]) < 0.06f)
{
waterError += d * d;
++waterN;
}
}
}
// The noisy input itself, nearest-upsampled: the error with no denoising and no reconstruction.
for (UINT y = 0; y < oh; ++y)
for (UINT x = 0; x < ow; ++x)
{
const size_t i = static_cast<size_t>(y) * ow + x;
const size_t s = static_cast<size_t>(y * rh / oh) * rw + x * rw / ow;
const V3 t = truth.colorClean[i];
const float tv[3] = {t.x, t.y, t.z};
for (int c = 0; c < 3; ++c)
{
const float v = std::max(0.0f, from_half(scene.color[s * 4 + c]));
const double d = v / (1 + v) - tv[c] / (1 + tv[c]);
noisyError += d * d;
}
}
const double rmse = n ? std::sqrt(error / n) : -1, noisyRmse = std::sqrt(noisyError / (3.0 * ow * oh));
sized = sized && unwritten == 0;
finite = finite && nonFinite == 0;
if (f + 1 == cutFrame)
errorBeforeCut = rmse;
if (f == cutFrame)
errorAtCut = rmse;
if (f + 1 == frames)
errorFinal = rmse;
const double psnr = rmse > 0 ? 20 * std::log10(1.0 / rmse) : 0;
const double waterRmse = waterN ? std::sqrt(waterError / waterN) : -1;
if (fsr_scene::g_water)
std::printf("frame %u: rmse %.5f water %.5f\n", f, rmse, waterRmse);
log += format("%s\n {\"frame\": %u, \"reset\": %s, \"result\": %u, \"rmse\": %.5f, \"psnr_db\": %.2f, \"water_rmse\": %.5f, "
"\"noisy_input_rmse\": %.5f, \"unwritten\": %zu, \"nonfinite\": %zu, \"call_us\": %.0f, "
"\"submit_wait_us\": %.0f}",
f ? "," : "", f, reset ? "true" : "false", evaluated, rmse, psnr, waterRmse, noisyRmse, unwritten, nonFinite,
callUs.back(), totalUs.back());
if (f == 0 || f + 1 == moveStart || f + 1 == cutFrame || f == cutFrame || f + 1 == frames)
{
write_ppm(format("%s\\frame%03u_output.ppm", outDir.c_str(), f), ow, oh, result);
write_ppm(format("%s\\frame%03u_truth.ppm", outDir.c_str(), f), ow, oh, truth.colorClean);
std::vector<V3> input(static_cast<size_t>(rw) * rh);
for (size_t i = 0; i < input.size(); ++i)
input[i] = {from_half(scene.color[i * 4]), from_half(scene.color[i * 4 + 1]), from_half(scene.color[i * 4 + 2])};
write_ppm(format("%s\\frame%03u_input.ppm", outDir.c_str(), f), rw, rh, input);
}
previous = camera;
}
json_field("frame_log", log + "\n ]");
const unsigned long long vramPeak = process_vram_kib();
std::vector<double> warmCall(callUs.begin() + std::min<size_t>(callUs.size(), 4), callUs.end());
std::vector<double> warmTotal(totalUs.begin() + std::min<size_t>(totalUs.size(), 4), totalUs.end());
std::sort(warmCall.begin(), warmCall.end());
std::sort(warmTotal.begin(), warmTotal.end());
const double medianCall = warmCall.empty() ? -1 : warmCall[warmCall.size() / 2];
const double medianTotal = warmTotal.empty() ? -1 : warmTotal[warmTotal.size() / 2];
json_field("timing_us", format("{\"evaluate_call_median\": %.0f, \"submit_wait_median\": %.0f, \"first_frame_submit_wait\": %.0f}",
medianCall, medianTotal, totalUs.empty() ? -1.0 : totalUs[0]));
json_field("vram_mib", format("{\"before_init\": %.1f, \"after_frames\": %.1f}", vramStart / 1024.0, vramPeak / 1024.0));
check("evaluations", allEvaluated, format("%u frames, EvaluateFeature call median %.0f us, call+submit+wait median %.0f us",
frames, medianCall, medianTotal));
check("output_size", sized, format("every texel of the %ux%u output written on every frame", ow, oh));
check("output_finite", finite, "no NaN/Inf");
json_field("rmse", format("{\"before_cut\": %.5f, \"at_cut\": %.5f, \"final\": %.5f}", errorBeforeCut, errorAtCut, errorFinal));
std::printf("tonemapped RMSE vs ground truth: before cut %.5f, at cut %.5f, final %.5f\n", errorBeforeCut, errorAtCut,
errorFinal);
const NgxResult released = release(feature);
// Let the shim's deferred cleanup see the list retire.
submit_and_wait();
const NgxResult shut = shutdown();
check("release", released == NGX_SUCCESS && shut == NGX_SUCCESS, format("release 0x%08x, shutdown 0x%08x", released, shut));
return finish(g_failed ? 1 : 0);
}
tools/vulkan_output_conversion_probe.cpp
+133
View File
@@ -0,0 +1,133 @@
// Compare fused storage output with the original RGBA16F copy + R11G11B10 blit.
#include "d4r_vulkan_staging.h"
#include <cassert>
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <vector>
static void check(VkResult r) { if (r != VK_SUCCESS) { std::fprintf(stderr,"Vulkan error %d\n",r); std::exit(1); } }
struct Image { VkImage image; VkImageView view; VkDeviceMemory memory; VkFormat format; bool depth; };
int main() {
constexpr uint32_t W=257, H=257;
constexpr size_t BYTES=W*H*16;
VkApplicationInfo app={VK_STRUCTURE_TYPE_APPLICATION_INFO}; app.apiVersion=VK_API_VERSION_1_1;
VkInstanceCreateInfo ic={VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO};ic.pApplicationInfo=&app;
VkInstance instance;check(vkCreateInstance(&ic,nullptr,&instance));
uint32_t count=0;check(vkEnumeratePhysicalDevices(instance,&count,nullptr));
std::vector<VkPhysicalDevice> devices(count);check(vkEnumeratePhysicalDevices(instance,&count,devices.data()));
VkPhysicalDevice physical=VK_NULL_HANDLE;
for(auto d:devices){VkPhysicalDeviceProperties p;vkGetPhysicalDeviceProperties(d,&p);if(p.vendorID==0x1002){physical=d;std::printf("GPU: %s\n",p.deviceName);break;}}
assert(physical);
vkGetPhysicalDeviceQueueFamilyProperties(physical,&count,nullptr);std::vector<VkQueueFamilyProperties> families(count);
vkGetPhysicalDeviceQueueFamilyProperties(physical,&count,families.data());
uint32_t family=0;while(family<count && !(families[family].queueFlags&VK_QUEUE_COMPUTE_BIT))++family;assert(family<count);
float priority=1;VkDeviceQueueCreateInfo qi={VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO};qi.queueFamilyIndex=family;qi.queueCount=1;qi.pQueuePriorities=&priority;
VkPhysicalDeviceFeatures supported, enabled={};vkGetPhysicalDeviceFeatures(physical,&supported);
enabled.shaderStorageImageExtendedFormats=supported.shaderStorageImageExtendedFormats;
VkDeviceCreateInfo dc={VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO};dc.queueCreateInfoCount=1;dc.pQueueCreateInfos=&qi;dc.pEnabledFeatures=&enabled;
VkDevice device;check(vkCreateDevice(physical,&dc,nullptr,&device));VkQueue queue;vkGetDeviceQueue(device,family,0,&queue);
VkPhysicalDeviceMemoryProperties memory;vkGetPhysicalDeviceMemoryProperties(physical,&memory);
auto allocate=[&](VkMemoryRequirements req,VkMemoryPropertyFlags flags){
uint32_t type=0;while(type<memory.memoryTypeCount && (!(req.memoryTypeBits&(1u<<type)) || (memory.memoryTypes[type].propertyFlags&flags)!=flags))++type;assert(type<memory.memoryTypeCount);
VkMemoryAllocateInfo a={VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO};a.allocationSize=req.size;a.memoryTypeIndex=type;VkDeviceMemory m;check(vkAllocateMemory(device,&a,nullptr,&m));return m;
};
auto image=[&](VkFormat format,bool output){
Image result={};result.format=format;result.depth=format==VK_FORMAT_D32_SFLOAT_S8_UINT;
VkImageCreateInfo info={VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO};info.imageType=VK_IMAGE_TYPE_2D;info.format=format;info.extent={W,H,1};info.mipLevels=info.arrayLayers=1;info.samples=VK_SAMPLE_COUNT_1_BIT;info.tiling=VK_IMAGE_TILING_OPTIMAL;
// Transfers build the reference and read back packed R11G11B10 words.
info.usage=VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT | (output ? VK_IMAGE_USAGE_STORAGE_BIT : 0);
check(vkCreateImage(device,&info,nullptr,&result.image));VkMemoryRequirements req;vkGetImageMemoryRequirements(device,result.image,&req);
result.memory=allocate(req,0);check(vkBindImageMemory(device,result.image,result.memory,0));
VkImageViewCreateInfo view={VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO};view.image=result.image;view.viewType=VK_IMAGE_VIEW_TYPE_2D;view.format=format;
view.subresourceRange={result.depth ? VK_IMAGE_ASPECT_DEPTH_BIT : VK_IMAGE_ASPECT_COLOR_BIT,0,1,0,1};
check(vkCreateImageView(device,&view,nullptr,&result.view));return result;
};
auto destroyImage=[&](Image& i){vkDestroyImageView(device,i.view,nullptr);vkDestroyImage(device,i.image,nullptr);vkFreeMemory(device,i.memory,nullptr);};
VkBufferCreateInfo bi={VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO};bi.size=BYTES;bi.usage=VK_BUFFER_USAGE_STORAGE_BUFFER_BIT|VK_BUFFER_USAGE_TRANSFER_SRC_BIT|VK_BUFFER_USAGE_TRANSFER_DST_BIT;
VkBuffer buffers[2];VkDeviceMemory allocations[2];float* mapped[2];
for(int i=0;i<2;++i){check(vkCreateBuffer(device,&bi,nullptr,&buffers[i]));VkMemoryRequirements req;vkGetBufferMemoryRequirements(device,buffers[i],&req);allocations[i]=allocate(req,VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT|VK_MEMORY_PROPERTY_HOST_COHERENT_BIT);check(vkBindBufferMemory(device,buffers[i],allocations[i],0));check(vkMapMemory(device,allocations[i],0,VK_WHOLE_SIZE,0,reinterpret_cast<void**>(&mapped[i])));}
D4rVkStaging staging;assert(staging.init(device,vkGetDeviceProcAddr));
VkCommandPoolCreateInfo pi={VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO};pi.queueFamilyIndex=family;VkCommandPool pool;check(vkCreateCommandPool(device,&pi,nullptr,&pool));
VkCommandBufferAllocateInfo ca={VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO};ca.commandPool=pool;ca.level=VK_COMMAND_BUFFER_LEVEL_PRIMARY;ca.commandBufferCount=1;VkCommandBuffer cmd;check(vkAllocateCommandBuffers(device,&ca,&cmd));
auto begin=[&](){check(vkResetCommandPool(device,pool,0));VkCommandBufferBeginInfo b={VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO};b.flags=VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT;check(vkBeginCommandBuffer(cmd,&b));};
auto submit=[&](){VkMemoryBarrier host={VK_STRUCTURE_TYPE_MEMORY_BARRIER};host.srcAccessMask=VK_ACCESS_SHADER_WRITE_BIT|VK_ACCESS_TRANSFER_WRITE_BIT;host.dstAccessMask=VK_ACCESS_HOST_READ_BIT;vkCmdPipelineBarrier(cmd,VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT|VK_PIPELINE_STAGE_TRANSFER_BIT,VK_PIPELINE_STAGE_HOST_BIT,0,1,&host,0,nullptr,0,nullptr);check(vkEndCommandBuffer(cmd));VkSubmitInfo s={VK_STRUCTURE_TYPE_SUBMIT_INFO};s.commandBufferCount=1;s.pCommandBuffers=&cmd;check(vkQueueSubmit(queue,1,&s,VK_NULL_HANDLE));check(vkQueueWaitIdle(queue));};
auto barrier=[&](const Image& i,VkImageLayout from,VkImageLayout to,VkAccessFlags src,VkAccessFlags dst){
VkImageMemoryBarrier b={VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER};b.image=i.image;b.oldLayout=from;b.newLayout=to;b.srcAccessMask=src;b.dstAccessMask=dst;b.srcQueueFamilyIndex=b.dstQueueFamilyIndex=VK_QUEUE_FAMILY_IGNORED;
b.subresourceRange={static_cast<VkImageAspectFlags>(i.depth ? VK_IMAGE_ASPECT_DEPTH_BIT|VK_IMAGE_ASPECT_STENCIL_BIT : VK_IMAGE_ASPECT_COLOR_BIT),0,1,0,1};vkCmdPipelineBarrier(cmd,VK_PIPELINE_STAGE_ALL_COMMANDS_BIT,VK_PIPELINE_STAGE_ALL_COMMANDS_BIT,0,0,nullptr,0,nullptr,1,&b);
};
auto source=image(VK_FORMAT_R16G16B16A16_SFLOAT,false);
auto reference=image(VK_FORMAT_B10G11R11_UFLOAT_PACK32,true);
auto candidate=image(VK_FORMAT_B10G11R11_UFLOAT_PACK32,true);
D4rVkStagingSet write;assert(staging.set(write,candidate.view,candidate.format,buffers[0],W*H*8,true));
auto* halves=reinterpret_cast<uint16_t*>(mapped[0]);
for(unsigned i=0;i<W*H;++i){halves[i*4]=i;halves[i*4+1]=i*73+31;halves[i*4+2]=i*193+57;halves[i*4+3]=0x3c00;}
std::memset(mapped[1],0xa5,BYTES);
begin();barrier(source,VK_IMAGE_LAYOUT_UNDEFINED,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,0,VK_ACCESS_TRANSFER_WRITE_BIT);
VkBufferImageCopy region={};region.imageSubresource={VK_IMAGE_ASPECT_COLOR_BIT,0,0,1};region.imageExtent={W,H,1};
vkCmdCopyBufferToImage(cmd,buffers[0],source.image,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,1,&region);
barrier(source,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,VK_ACCESS_TRANSFER_WRITE_BIT,VK_ACCESS_TRANSFER_READ_BIT);
barrier(reference,VK_IMAGE_LAYOUT_UNDEFINED,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,0,VK_ACCESS_TRANSFER_WRITE_BIT);
VkImageBlit blit={};blit.srcSubresource=blit.dstSubresource=region.imageSubresource;
blit.srcOffsets[1]=blit.dstOffsets[1]={(int)W,(int)H,1};
vkCmdBlitImage(cmd,source.image,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,reference.image,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,1,&blit,VK_FILTER_NEAREST);
barrier(reference,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,VK_ACCESS_TRANSFER_WRITE_BIT,VK_ACCESS_TRANSFER_READ_BIT);
vkCmdCopyImageToBuffer(cmd,reference.image,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,buffers[1],1,&region);
barrier(candidate,VK_IMAGE_LAYOUT_UNDEFINED,VK_IMAGE_LAYOUT_GENERAL,0,VK_ACCESS_SHADER_WRITE_BIT);
staging.record(cmd,write,W,H);
barrier(candidate,VK_IMAGE_LAYOUT_GENERAL,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,VK_ACCESS_SHADER_WRITE_BIT,VK_ACCESS_TRANSFER_READ_BIT);
region.bufferOffset=W*H*4;
vkCmdCopyImageToBuffer(cmd,candidate.image,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,buffers[1],1,&region);submit();
auto* words=reinterpret_cast<uint32_t*>(mapped[1]);unsigned mismatches=0,finite=0,nan=0;
for(unsigned i=0;i<W*H;++i)for(unsigned c=0;c<3;++c) {
const auto h=halves[i*4+c];const unsigned shift=c*11,bits=c==2 ? 10 : 11,mask=(1u<<bits)-1;
unsigned old=(words[i]>>shift)&mask,now=(words[W*H+i]>>shift)&mask;
if((h&0x7c00)==0x7c00 && (h&0x3ff)) {
++nan;const unsigned m=c==2 ? 5 : 6;
assert((old>>m)==31 && (now>>m)==31 && (old&((1u<<m)-1)) && (now&((1u<<m)-1)));
} else {++finite;if(old!=now){if(mismatches<8)std::printf("DIFF pixel=%u channel=%u half=%04x blit=%x compute=%x\n",i,c,h,old,now);++mismatches;}}
}
std::printf("R11 output blit vs compute: finite_or_inf_channels=%u mismatches=%u nan_channels=%u\n",finite,mismatches,nan);
assert(mismatches==0);
staging.free(write);destroyImage(source);destroyImage(reference);destroyImage(candidate);
// Input conversion: every unsigned R11/R10 channel encoding, padded rows.
auto input=image(VK_FORMAT_B10G11R11_UFLOAT_PACK32,false);
auto input_reference=image(VK_FORMAT_R16G16B16A16_SFLOAT,true);
D4rVkStagingSet read;assert(staging.set(read,input.view,input.format,buffers[0],BYTES,false));
auto* raw=reinterpret_cast<uint32_t*>(mapped[0]);
for(unsigned i=0;i<BYTES/4;++i)raw[i]=0xa5a5a5a5;
for(unsigned i=0;i<W*H;++i)raw[i]=(i&2047u)|((i*73u+31u)&2047u)<<11|((i*193u+57u)&1023u)<<22;
std::vector<uint32_t> initial(raw,raw+BYTES/4);
begin();barrier(input,VK_IMAGE_LAYOUT_UNDEFINED,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,0,VK_ACCESS_TRANSFER_WRITE_BIT);
region.bufferOffset=0;
vkCmdCopyBufferToImage(cmd,buffers[0],input.image,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,1,&region);
barrier(input,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,VK_ACCESS_TRANSFER_WRITE_BIT,VK_ACCESS_TRANSFER_READ_BIT);
barrier(input_reference,VK_IMAGE_LAYOUT_UNDEFINED,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,0,VK_ACCESS_TRANSFER_WRITE_BIT);
vkCmdBlitImage(cmd,input.image,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,input_reference.image,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,1,&blit,VK_FILTER_NEAREST);
barrier(input_reference,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,VK_ACCESS_TRANSFER_WRITE_BIT,VK_ACCESS_TRANSFER_READ_BIT);
vkCmdCopyImageToBuffer(cmd,input_reference.image,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,buffers[1],1,&region);
barrier(input,VK_IMAGE_LAYOUT_TRANSFER_SRC_OPTIMAL,VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,VK_ACCESS_TRANSFER_READ_BIT,VK_ACCESS_SHADER_READ_BIT);
VkMemoryBarrier reuse={VK_STRUCTURE_TYPE_MEMORY_BARRIER};reuse.srcAccessMask=VK_ACCESS_TRANSFER_READ_BIT;reuse.dstAccessMask=VK_ACCESS_SHADER_WRITE_BIT;
vkCmdPipelineBarrier(cmd,VK_PIPELINE_STAGE_TRANSFER_BIT,VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT,0,1,&reuse,0,nullptr,0,nullptr);
constexpr unsigned stride=W+7;
staging.record(cmd,read,W,H,1,stride);submit();
const auto* actual=reinterpret_cast<const uint16_t*>(mapped[0]);
const auto* expected=reinterpret_cast<const uint16_t*>(mapped[1]);
unsigned input_bad=0,input_nan=0;
for(unsigned y=0;y<H;++y) {
for(unsigned x=0;x<W;++x)for(unsigned c=0;c<4;++c) {
const auto got=actual[(size_t(y)*stride+x)*4+c],want=expected[(size_t(y)*W+x)*4+c];
if((want&0x7c00)==0x7c00 && (want&0x3ff)) {
++input_nan;assert((got&0x7c00)==0x7c00 && (got&0x3ff));
} else if(got!=want){if(input_bad<8)std::printf("INPUT DIFF x=%u y=%u channel=%u blit=%04x compute=%04x\n",x,y,c,want,got);++input_bad;}
}
for(unsigned word=(y*stride+W)*2;word<(y+1)*stride*2;++word)assert(raw[word]==initial[word]);
}
for(unsigned word=H*stride*2;word<BYTES/4;++word)assert(raw[word]==initial[word]);
std::printf("R11 input blit vs compute: mismatches=%u nan_channels=%u padded_rows_unchanged=1\n",input_bad,input_nan);
assert(input_bad==0);
staging.free(read);destroyImage(input);destroyImage(input_reference);
staging.destroy();vkDestroyCommandPool(device,pool,nullptr);
for(int i=0;i<2;++i){vkUnmapMemory(device,allocations[i]);vkDestroyBuffer(device,buffers[i],nullptr);vkFreeMemory(device,allocations[i],nullptr);}
vkDestroyDevice(device,nullptr);vkDestroyInstance(instance,nullptr);std::puts("PASS: exhaustive half inputs, all RGB channels, partial workgroups");
}
tools/wine_nvcuda_bridge.c
+346 -3
View File
@@ -14,6 +14,8 @@
#include <time.h>
#include "d4r_native_selection.h"
#include "d4r_motion_dilation.h"
#include "d4r_native_motion.h"
#include "d4r_wire_history.h"
#ifdef D4R_MOTION_KERNELS
#include "d4r_motion_kernels.h"
#endif
@@ -983,9 +985,11 @@ static void* find_zluda_symbol(const char* name)
return cuda_library != NULL ? dlsym(cuda_library, name) : NULL;
}
/* d4r-owned, opt-in preprocessing. Runs on HIP's legacy/default stream, shared
with CUDA's null stream; the shim records and waits for completion before NGX. */
CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters)
static CUstream d4r_ngx_stream(CUstream stream);
/* Preprocessing, input waits and NGX null-stream launches all use d4r_ngx_stream.
allocation_height == 0 retains the original ABI's active-height contract. */
static CUresult dilate_motion_vectors(const D4rMotionDilationParams* parameters, uint32_t allocation_height, int thin_sky)
{
if (parameters == NULL || parameters->depth == 0 || parameters->motion == 0 || parameters->output == 0 ||
parameters->radius < 1 || parameters->radius > 2 || parameters->render_width == 0 ||
@@ -1002,6 +1006,8 @@ CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters
static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
static void* module;
static void* function;
static void* lds_function;
static void* thin_function;
static HIP_LAUNCH launch;
static int initialized, result = CUDA_ERROR_NOT_SUPPORTED;
pthread_mutex_lock(&lock);
@@ -1018,7 +1024,11 @@ CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters
launch = hip != NULL ? (HIP_LAUNCH)dlsym(hip, "hipModuleLaunchKernel") : NULL;
if (image != NULL && load != NULL && get != NULL && launch != NULL &&
load(&module, image) == 0 && get(&function, module, "d4r_motion_dilate") == 0)
{
result = CUDA_SUCCESS;
get(&lds_function, module, "d4r_motion_dilate_lds");
get(&thin_function, module, "d4r_motion_dilate_lds_thin");
}
tracef("motion-vector dilation: %s (%s)", result == 0 ? "GPU kernel ready" : "unavailable", architecture);
}
const int status = result;
@@ -1026,6 +1036,19 @@ CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters
if (status != CUDA_SUCCESS)
return status;
D4rMotionDilationParams copy = *parameters;
const int lds = lds_function != NULL && d4r_motion_dilation_lds_eligible(&copy);
if (allocation_height != 0 && (!lds || allocation_height < copy.motion_height ||
copy.motion_pitch % 4 != 0 || copy.motion_pitch / 4 > 0x7fffffffu || allocation_height > 0x7fffffffu))
return CUDA_ERROR_INVALID_VALUE;
if (thin_sky && (!lds || thin_function == NULL))
return CUDA_ERROR_NOT_SUPPORTED;
if (lds)
{
uint32_t height = allocation_height != 0 ? allocation_height : copy.motion_height;
void* arguments[] = {&copy, &height};
return launch(thin_sky ? thin_function : lds_function, (unsigned)((copy.motion_pitch / 4 + 15) / 16), (height + 15) / 16,
1, 256, 1, 1, 0, d4r_ngx_stream(NULL), arguments, NULL);
}
void* arguments[] = {&copy};
const uint64_t pixels = (uint64_t)copy.motion_width * copy.motion_height;
return launch(function, (unsigned)((pixels + 127) / 128), 1, 1, 128, 1, 1, 0, d4r_ngx_stream(NULL), arguments, NULL);
@@ -1034,6 +1057,326 @@ CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters
#endif
}
CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters)
{
return dilate_motion_vectors(parameters, 0, 0);
}
/* Additive export: retain the 72-byte parameter ABI and preserve the full padded
allocation in the LDS launch, without a separate full-plane copy. */
CUresult WINAPI d4rDilateMotionVectorsPadded(const D4rMotionDilationParams* parameters, uint32_t allocation_height)
{
if (allocation_height == 0)
return CUDA_ERROR_INVALID_VALUE;
return dilate_motion_vectors(parameters, allocation_height, 0);
}
CUresult WINAPI d4rDilateMotionVectorsThin(const D4rMotionDilationParams* parameters, uint32_t allocation_height)
{
if (allocation_height == 0)
return CUDA_ERROR_INVALID_VALUE;
return dilate_motion_vectors(parameters, allocation_height, 1);
}
/* Additive native-MV operation. The recorded input marker must precede this
* preprocessing launch, and NGX follows it on the same effective stream. */
static CUresult d4r_compose_native_motion(const D4rNativeMotionParams* parameters, unsigned int thin,float jitter_x,float jitter_y)
{
if(!d4r_native_motion_parameters_valid(parameters)) return CUDA_ERROR_INVALID_VALUE;
#ifdef D4R_MOTION_KERNELS
typedef int(__attribute__((sysv_abi)) * HIP_LOAD)(void**,const void*);
typedef int(__attribute__((sysv_abi)) * HIP_FUNCTION)(void**,void*,const char*);
typedef int(__attribute__((sysv_abi)) * HIP_LAUNCH)(void*,unsigned,unsigned,unsigned,
unsigned,unsigned,unsigned,unsigned,void*,void**,void**);
static pthread_mutex_t lock=PTHREAD_MUTEX_INITIALIZER;
static void *module,*function[3];
static HIP_LAUNCH launch;
static int initialized,result=CUDA_ERROR_NOT_SUPPORTED;
int status;
pthread_mutex_lock(&lock);
if(!initialized)
{
void* hip;
const void* image;
HIP_LOAD load;
HIP_FUNCTION get;
char architecture[32];
initialized=1;pthread_once(&load_once,load_zluda);
hip=dlopen("libamdhip64.so.7",RTLD_NOW|RTLD_LOCAL);
gpu_architecture(architecture,sizeof(architecture));image=d4r_motion_image(architecture);
load=hip ? (HIP_LOAD)dlsym(hip,"hipModuleLoadData") : NULL;
get=hip ? (HIP_FUNCTION)dlsym(hip,"hipModuleGetFunction") : NULL;
launch=hip ? (HIP_LAUNCH)dlsym(hip,"hipModuleLaunchKernel") : NULL;
if(image && load && get && launch && load(&module,image)==0 &&
get(&function[0],module,"d4r_native_motion_compose")==0 &&
get(&function[1],module,"d4r_native_motion_compose_thin")==0 &&
get(&function[2],module,"d4r_native_motion_compose_aligned")==0) result=CUDA_SUCCESS;
tracef("native motion composition: %s (%s)",result==0 ? "GPU kernel ready" : "unavailable",architecture);
}
status=result;pthread_mutex_unlock(&lock);
if(status==CUDA_SUCCESS)
{
D4rNativeMotionThinParams copy={*parameters,jitter_x,jitter_y};
void* arguments[]={&copy};
return launch(function[thin],(copy.base.width+15)/16,(copy.base.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
}
return status;
#else
return CUDA_ERROR_NOT_SUPPORTED;
#endif
}
CUresult WINAPI d4rComposeNativeMotion(const D4rNativeMotionParams* parameters)
{ return d4r_compose_native_motion(parameters,0,0,0); }
CUresult WINAPI d4rComposeNativeMotionThin(const D4rNativeMotionThinParams* parameters)
{
if(!parameters || !(parameters->jitter_x>=-1 && parameters->jitter_x<=1 && parameters->jitter_y>=-1 && parameters->jitter_y<=1)) return CUDA_ERROR_INVALID_VALUE;
return d4r_compose_native_motion(&parameters->base,1,parameters->jitter_x,parameters->jitter_y);
}
CUresult WINAPI d4rComposeNativeMotionAligned(const D4rNativeMotionThinParams* parameters)
{
if(!parameters || !(parameters->jitter_x>=-1 && parameters->jitter_x<=1 && parameters->jitter_y>=-1 && parameters->jitter_y<=1)) return CUDA_ERROR_INVALID_VALUE;
return d4r_compose_native_motion(&parameters->base,2,parameters->jitter_x,parameters->jitter_y);
}
CUresult WINAPI d4rStabilizeThinWires(const D4rWireHistoryParams* parameters)
{
if(!d4r_wire_history_parameters_valid(parameters)) return CUDA_ERROR_INVALID_VALUE;
#ifdef D4R_MOTION_KERNELS
typedef int(__attribute__((sysv_abi)) * HIP_LOAD)(void**,const void*);
typedef int(__attribute__((sysv_abi)) * HIP_FUNCTION)(void**,void*,const char*);
typedef int(__attribute__((sysv_abi)) * HIP_LAUNCH)(void*,unsigned,unsigned,unsigned,
unsigned,unsigned,unsigned,unsigned,void*,void**,void**);
static pthread_mutex_t lock=PTHREAD_MUTEX_INITIALIZER;
static void *module,*function;
static HIP_LAUNCH launch;
static int initialized,result=CUDA_ERROR_NOT_SUPPORTED;
int status;
pthread_mutex_lock(&lock);
if(!initialized)
{
void* hip;
const void* image;
HIP_LOAD load;
HIP_FUNCTION get;
char architecture[32];
initialized=1;pthread_once(&load_once,load_zluda);
hip=dlopen("libamdhip64.so.7",RTLD_NOW|RTLD_LOCAL);
gpu_architecture(architecture,sizeof(architecture));image=d4r_motion_image(architecture);
load=hip ? (HIP_LOAD)dlsym(hip,"hipModuleLoadData") : NULL;
get=hip ? (HIP_FUNCTION)dlsym(hip,"hipModuleGetFunction") : NULL;
launch=hip ? (HIP_LAUNCH)dlsym(hip,"hipModuleLaunchKernel") : NULL;
if(image && load && get && launch && load(&module,image)==0 &&
get(&function,module,"d4r_wire_history")==0) result=CUDA_SUCCESS;
tracef("thin wire history: %s (%s)",result==0 ? "GPU kernel ready" : "unavailable",architecture);
}
status=result;pthread_mutex_unlock(&lock);
if(status==CUDA_SUCCESS)
{
D4rWireHistoryParams copy=*parameters;
void* arguments[]={&copy};
return launch(function,(copy.width+15)/16,(copy.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
}
return status;
#else
return CUDA_ERROR_NOT_SUPPORTED;
#endif
}
CUresult WINAPI d4rComposeNativeMotionHybrid(const D4rNativeMotionHybridParams* parameters)
{
if(!d4r_native_motion_hybrid_parameters_valid(parameters)) return CUDA_ERROR_INVALID_VALUE;
#ifdef D4R_MOTION_KERNELS
typedef int(__attribute__((sysv_abi)) * HIP_LOAD)(void**,const void*);
typedef int(__attribute__((sysv_abi)) * HIP_FUNCTION)(void**,void*,const char*);
typedef int(__attribute__((sysv_abi)) * HIP_LAUNCH)(void*,unsigned,unsigned,unsigned,
unsigned,unsigned,unsigned,unsigned,void*,void**,void**);
static pthread_mutex_t lock=PTHREAD_MUTEX_INITIALIZER;
static void *module,*function;
static HIP_LAUNCH launch;
static int initialized,result=CUDA_ERROR_NOT_SUPPORTED;
int status;
pthread_mutex_lock(&lock);
if(!initialized)
{
void* hip;
const void* image;
HIP_LOAD load;
HIP_FUNCTION get;
char architecture[32];
initialized=1;pthread_once(&load_once,load_zluda);
hip=dlopen("libamdhip64.so.7",RTLD_NOW|RTLD_LOCAL);
gpu_architecture(architecture,sizeof(architecture));image=d4r_motion_image(architecture);
load=hip ? (HIP_LOAD)dlsym(hip,"hipModuleLoadData") : NULL;
get=hip ? (HIP_FUNCTION)dlsym(hip,"hipModuleGetFunction") : NULL;
launch=hip ? (HIP_LAUNCH)dlsym(hip,"hipModuleLaunchKernel") : NULL;
if(image && load && get && launch && load(&module,image)==0 &&
get(&function,module,"d4r_native_motion_compose_hybrid")==0) result=CUDA_SUCCESS;
tracef("hybrid native motion: %s (%s)",result==0 ? "GPU kernel ready" : "unavailable",architecture);
}
status=result;pthread_mutex_unlock(&lock);
if(status==CUDA_SUCCESS)
{
D4rNativeMotionHybridParams copy=*parameters;
void* arguments[]={&copy};
return launch(function,(copy.native.base.width+15)/16,(copy.native.base.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
}
return status;
#else
return CUDA_ERROR_NOT_SUPPORTED;
#endif
}
CUresult WINAPI d4rStabilizeThinWiresSparse(const D4rWireHistorySparseParams* parameters)
{
if(!d4r_wire_history_sparse_parameters_valid(parameters)) return CUDA_ERROR_INVALID_VALUE;
#ifdef D4R_MOTION_KERNELS
typedef int(__attribute__((sysv_abi)) * HIP_LOAD)(void**,const void*);
typedef int(__attribute__((sysv_abi)) * HIP_FUNCTION)(void**,void*,const char*);
typedef int(__attribute__((sysv_abi)) * HIP_LAUNCH)(void*,unsigned,unsigned,unsigned,
unsigned,unsigned,unsigned,unsigned,void*,void**,void**);
static pthread_mutex_t lock=PTHREAD_MUTEX_INITIALIZER;
static void *module,*function,*apply;
static HIP_LAUNCH launch;
static int initialized,result=CUDA_ERROR_NOT_SUPPORTED;
int status;
pthread_mutex_lock(&lock);
if(!initialized)
{
void* hip;
const void* image;
HIP_LOAD load;
HIP_FUNCTION get;
char architecture[32];
initialized=1;pthread_once(&load_once,load_zluda);
hip=dlopen("libamdhip64.so.7",RTLD_NOW|RTLD_LOCAL);
gpu_architecture(architecture,sizeof(architecture));image=d4r_motion_image(architecture);
load=hip ? (HIP_LOAD)dlsym(hip,"hipModuleLoadData") : NULL;
get=hip ? (HIP_FUNCTION)dlsym(hip,"hipModuleGetFunction") : NULL;
launch=hip ? (HIP_LAUNCH)dlsym(hip,"hipModuleLaunchKernel") : NULL;
if(image && load && get && launch && load(&module,image)==0 &&
get(&function,module,"d4r_wire_history_sparse")==0 &&
get(&apply,module,"d4r_wire_history_apply")==0) result=CUDA_SUCCESS;
tracef("sparse thin wire history: %s (%s)",result==0 ? "GPU kernel ready" : "unavailable",architecture);
}
status=result;pthread_mutex_unlock(&lock);
if(status==CUDA_SUCCESS)
{
D4rWireHistorySparseParams copy=*parameters;
void* arguments[]={&copy};
status=launch(function,(copy.base.width+15)/16,(copy.base.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
if(status!=0) return status;
return launch(apply,(copy.base.width+15)/16,(copy.base.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
}
return status;
#else
return CUDA_ERROR_NOT_SUPPORTED;
#endif
}
CUresult WINAPI d4rComposeNativeMotionPrepared(const D4rNativeMotionPreparedParams* parameters)
{
if(!d4r_native_motion_prepared_parameters_valid(parameters)) return CUDA_ERROR_INVALID_VALUE;
#ifdef D4R_MOTION_KERNELS
typedef int(__attribute__((sysv_abi)) * HIP_LOAD)(void**,const void*);
typedef int(__attribute__((sysv_abi)) * HIP_FUNCTION)(void**,void*,const char*);
typedef int(__attribute__((sysv_abi)) * HIP_LAUNCH)(void*,unsigned,unsigned,unsigned,
unsigned,unsigned,unsigned,unsigned,void*,void**,void**);
static pthread_mutex_t lock=PTHREAD_MUTEX_INITIALIZER;
static void *module,*function,*apply;
static HIP_LAUNCH launch;
static int initialized,result=CUDA_ERROR_NOT_SUPPORTED;
int status;
pthread_mutex_lock(&lock);
if(!initialized)
{
void* hip;
const void* image;
HIP_LOAD load;
HIP_FUNCTION get;
char architecture[32];
initialized=1;pthread_once(&load_once,load_zluda);
hip=dlopen("libamdhip64.so.7",RTLD_NOW|RTLD_LOCAL);
gpu_architecture(architecture,sizeof(architecture));image=d4r_motion_image(architecture);
load=hip ? (HIP_LOAD)dlsym(hip,"hipModuleLoadData") : NULL;
get=hip ? (HIP_FUNCTION)dlsym(hip,"hipModuleGetFunction") : NULL;
launch=hip ? (HIP_LAUNCH)dlsym(hip,"hipModuleLaunchKernel") : NULL;
if(image && load && get && launch && load(&module,image)==0 &&
get(&function,module,"d4r_native_motion_prepare")==0 &&
get(&apply,module,"d4r_native_motion_prepared")==0) result=CUDA_SUCCESS;
tracef("prepared native motion: %s (%s)",result==0 ? "GPU kernel ready" : "unavailable",architecture);
}
status=result;pthread_mutex_unlock(&lock);
if(status==CUDA_SUCCESS)
{
D4rNativeMotionPreparedParams copy=*parameters;
void* arguments[]={&copy};
status=launch(function,(copy.native.base.width+31)/32,(copy.native.base.height+31)/32,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
if(status!=0) return status;
return launch(apply,(copy.native.base.width+15)/16,(copy.native.base.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
}
return status;
#else
return CUDA_ERROR_NOT_SUPPORTED;
#endif
}
CUresult WINAPI d4rStabilizeThinWiresPrepared(const D4rWireHistoryPreparedParams* parameters)
{
if(!d4r_wire_history_prepared_parameters_valid(parameters)) return CUDA_ERROR_INVALID_VALUE;
#ifdef D4R_MOTION_KERNELS
typedef int(__attribute__((sysv_abi)) * HIP_LOAD)(void**,const void*);
typedef int(__attribute__((sysv_abi)) * HIP_FUNCTION)(void**,void*,const char*);
typedef int(__attribute__((sysv_abi)) * HIP_LAUNCH)(void*,unsigned,unsigned,unsigned,
unsigned,unsigned,unsigned,unsigned,void*,void**,void**);
static pthread_mutex_t lock=PTHREAD_MUTEX_INITIALIZER;
static void *module,*function,*apply;
static HIP_LAUNCH launch;
static int initialized,result=CUDA_ERROR_NOT_SUPPORTED;
int status;
pthread_mutex_lock(&lock);
if(!initialized)
{
void* hip;
const void* image;
HIP_LOAD load;
HIP_FUNCTION get;
char architecture[32];
initialized=1;pthread_once(&load_once,load_zluda);
hip=dlopen("libamdhip64.so.7",RTLD_NOW|RTLD_LOCAL);
gpu_architecture(architecture,sizeof(architecture));image=d4r_motion_image(architecture);
load=hip ? (HIP_LOAD)dlsym(hip,"hipModuleLoadData") : NULL;
get=hip ? (HIP_FUNCTION)dlsym(hip,"hipModuleGetFunction") : NULL;
launch=hip ? (HIP_LAUNCH)dlsym(hip,"hipModuleLaunchKernel") : NULL;
if(image && load && get && launch && load(&module,image)==0 &&
get(&function,module,"d4r_wire_history_preclassified")==0 &&
get(&apply,module,"d4r_wire_history_apply")==0) result=CUDA_SUCCESS;
tracef("prepared thin wire history: %s (%s)",result==0 ? "GPU kernel ready" : "unavailable",architecture);
}
status=result;pthread_mutex_unlock(&lock);
if(status==CUDA_SUCCESS)
{
D4rWireHistoryPreparedParams copy=*parameters;
void* arguments[]={&copy};
status=launch(function,(copy.sparse.base.width+15)/16,(copy.sparse.base.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
if(status!=0) return status;
return launch(apply,(copy.sparse.base.width+15)/16,(copy.sparse.base.height+15)/16,1,256,1,1,0,
d4r_ngx_stream(NULL),arguments,NULL);
}
return status;
#else
return CUDA_ERROR_NOT_SUPPORTED;
#endif
}
static void tracef(const char* format, ...)
{
char line[2048];
tools/wine_nvcuda_bridge.spec
+16
View File
@@ -125,3 +125,19 @@
@ stdcall d4rNgxTimelineDestroy(ptr) d4rNgxTimelineDestroy
@ stdcall d4rNgxTimelineWait(ptr) d4rNgxTimelineWait
@ stdcall d4rDilateMotionVectors(ptr) d4rDilateMotionVectors
@ stdcall d4rDilateMotionVectorsPadded(ptr long) d4rDilateMotionVectorsPadded
@ stdcall d4rDilateMotionVectorsThin(ptr long) d4rDilateMotionVectorsThin
@ stdcall d4rComposeNativeMotion(ptr) d4rComposeNativeMotion
@ stdcall d4rComposeNativeMotionThin(ptr) d4rComposeNativeMotionThin
@ stdcall d4rStabilizeThinWires(ptr) d4rStabilizeThinWires
@ stdcall d4rStabilizeThinWiresSparse(ptr) d4rStabilizeThinWiresSparse
@ stdcall d4rComposeNativeMotionAligned(ptr) d4rComposeNativeMotionAligned
@ stdcall d4rComposeNativeMotionHybrid(ptr) d4rComposeNativeMotionHybrid
@ stdcall d4rComposeNativeMotionPrepared(ptr) d4rComposeNativeMotionPrepared
@ stdcall d4rStabilizeThinWiresPrepared(ptr) d4rStabilizeThinWiresPrepared
vulkan/kvk.cpp
+7
View File
@@ -90,6 +90,13 @@ int main(int argc, char** argv)
float prio = 1; VkDeviceQueueCreateInfo qci{VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO}; qci.queueFamilyIndex = qi; qci.queueCount = 1; qci.pQueuePriorities = &prio;
std::vector<const char*> exts;
if (cm.cooperativeMatrix) exts.push_back(VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME);
// Some drivers report shaderFloat8 without exposing the extension (RADV gfx11). Only enable an advertised extension.
uint32_t ne = 0; CK(vkEnumerateDeviceExtensionProperties(pd, nullptr, &ne, nullptr));
std::vector<VkExtensionProperties> ep(ne); CK(vkEnumerateDeviceExtensionProperties(pd, nullptr, &ne, ep.data()));
const bool hasFloat8 = std::any_of(ep.begin(), ep.end(), [](const auto& e) {
return strcmp(e.extensionName, VK_EXT_SHADER_FLOAT8_EXTENSION_NAME) == 0;
});
if (!hasFloat8) f8.shaderFloat8 = f8.shaderFloat8CooperativeMatrix = VK_FALSE;
if (f8.shaderFloat8) exts.push_back(VK_EXT_SHADER_FLOAT8_EXTENSION_NAME);
if (!cm.cooperativeMatrix) v13.pNext = nullptr; else if (!f8.shaderFloat8) cm.pNext = nullptr;
VkDeviceCreateInfo dci{VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO, &f2}; dci.queueCreateInfoCount = 1; dci.pQueueCreateInfos = &qci;