Improve GPU compatibility, RDNA2 kernels and startup diagnostics
- Match render/compute adapters and gate RR models by GPU capabilities - Add RDNA2 native texture kernels and fix FP16 rounding - Harden model builds, cache validation and packaging - Add startup tracing, timeouts, documentation and regression tests
This commit is contained in:
ayo committed
1 parent
ca93d0fbf7
commit
2558fcbd39
32 files changed
+1131
-81
No files matched your search
@@ -112,6 +112,8 @@ OptiScaler 0.9.4 checks a relative `NvngxPath` against the process working direc
|
||||
|
||||
## Logs
|
||||
|
||||
For startup stalls, GPU resets and model-build failures, see [Linux startup diagnostics](linux-startup-diagnostics.md).
|
||||
|
||||
The launcher writes each run's logs to `~/.cache/d4r-dlss-captures/game-<exe>-<time>/`. `d4r_nvngx.log` holds the shim's log. With `[Debug] Profile = true` it measures stage timings every frame and records sampled `D4R_PROFILE` lines. `ProfileLogEvery = 60` (the default) logs the first five frames and then every 60th frame; set it to `1` to log every frame, or another positive integer for a different interval. Restart the game after changing it. `D4R_PROFILE_LOG_EVERY` overrides the INI setting. FSR RR stage summaries average that many completed GPU samples; native Vulkan profiling uses the same log interval. Sampling reduces log writes while retaining profiling measurements and per-frame throughput intervals. `D4R_PROFILE_LOG=0` suppresses timing rows and FSR RR summaries.
|
||||
|
||||
Output completion uses event queries with 200 us NtDelayExecution sleeps by default. `D4R_SHIM_BLOCKING_SYNC = 0`
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
# Linux startup and GPU compatibility diagnostics
|
||||
|
||||
Keep one complete runtime together when comparing GPUs: the shim, nvcuda bridge, ZLUDA, patched D3D12 pair and
|
||||
native kernels. New shim/bridge logs include a source build identifier. Record file SHA-256 hashes as well when
|
||||
reporting failures; an old log format is not a controlled comparison with a new build.
|
||||
|
||||
## Capture a failing run
|
||||
|
||||
Add these entries to the existing `[Env]` section of `d4r/d4r.ini` (or add that section once):
|
||||
|
||||
```ini
|
||||
[Env]
|
||||
D4R_CUDA_VERBOSE = 1
|
||||
D4R_CUDA_TRACE = Z:\tmp\d4r-cuda-trace.log
|
||||
```
|
||||
|
||||
Use no shell quotes in INI values. This saves the bridge trace to `/tmp/d4r-cuda-trace.log` on the machine running
|
||||
the game. The bridge also writes to stderr. ZLUDA's own compiler/native-override messages require Proton's stderr
|
||||
log separately. Verbose tracing can affect timing; remove it after collecting the run.
|
||||
|
||||
Save the CUDA trace and `d4r_nvngx.log` separately for each successful/failed launch. The trace appends, while the
|
||||
default shim log is replaced on a fresh process. For a driver recovery or corrupted display, also save the kernel
|
||||
journal covering that run:
|
||||
|
||||
```sh
|
||||
journalctl -b -k -o short-monotonic --no-pager > d4r-kernel.log
|
||||
```
|
||||
|
||||
`CUDA_CREATE_BEGIN`, `CUDA_CREATE_WORKER_BEGIN`, `CUDA_CREATE_NGX_BEGIN/END` and `CUDA_CREATE_END` bracket feature
|
||||
creation. A worker still waiting logs progress every 15 seconds without abandoning resources still in use.
|
||||
`CUDA_BEGIN api=...` is recorded before module loading and, with verbose tracing, uploads, allocations, launches
|
||||
and synchronization. The following ordinary API line records its result. A launch return proves submission, not
|
||||
correct pixels or completed GPU work. Bridge records include a monotonic timestamp and thread ID.
|
||||
|
||||
## GPU selection
|
||||
|
||||
The game adapter's PCI address is matched to the compute device. Other compute devices are not given its LUID,
|
||||
and native kernels use the matching KFD topology entry rather than the largest GPU. When no PCI mapping is
|
||||
available, one visible compute GPU is unambiguous; multiple devices are refused instead of guessed. Selection
|
||||
failures report the rendering PCI address and visible compute devices. `D4R_GPU_ARCH` cannot override a known
|
||||
target with an incompatible architecture.
|
||||
|
||||
RDNA2 keeps ZLUDA's translation fallback and its existing software-matrix K kernels. It is not assigned a native
|
||||
FP8/WMMA Ray Reconstruction model. This does not establish that every translated model is stable or fast there.
|
||||
|
||||
## Model installer
|
||||
|
||||
The installer builds portable FP8 RR-D/E/F models plus separate `rr-d-gfx11` and `rr-e-gfx11` variants for compatible
|
||||
RDNA3 devices. The runtime checks model arithmetic and physical matrix capabilities before constructing RR
|
||||
pipelines. Unsupported models fall back to the existing CUDA path; advertised NGX availability is not a runtime
|
||||
validation. Engine `auto` has the same model-directory detection in the portable shim and Python launcher.
|
||||
|
||||
Model build logs are printed as they start and progress every 15 seconds. Each child has a 900-second deadline;
|
||||
`D4R_MODEL_TIMEOUT=SECONDS` changes it. The capture harness also limits individual GPU fence waits to 60 seconds
|
||||
(`D4R_HARNESS_GPU_TIMEOUT_MS`, 1–600000, changes that bound). A failed process or incomplete/nonfinite capture
|
||||
cannot be accepted solely because a replay filename exists.
|
||||
|
||||
Diagnostics are retained under `${XDG_CACHE_HOME:-$HOME/.cache}/d4r/model-builds/build.*`; `D4R_MODEL_LOG_DIR`
|
||||
changes that root. Successful builds retain logs and remove temporary weights/captures/prefixes. Failed builds
|
||||
retain their workspace, including proprietary model data; keep those artifacts local and remove them when no
|
||||
longer needed. Model folders carry `files.sha256.json`; cached builds are checked before reuse, and compiler
|
||||
arithmetic options participate in the cache key. Old caches without these manifests rebuild once.
|
||||
|
||||
These checks prevent known selection/setup failures and improve fault localization. They do not by themselves
|
||||
prove that an intermittent GPU reset or corrupted frame is fixed; that requires matching traces and GPU testing.
|
||||
@@ -44,7 +44,7 @@ The release is built this way, with hashes from DLSS 310.7.0 and 310.9.1. The PT
|
||||
- dec5 runs as four dependent phase kernels (`pwin_phase.h`, post kernels, needs patch 0008).
|
||||
- enc1 keeps one LDS buffer and reloads its input for the residual (`pwin_pos_recompute.h`); dec1 computes its Q, K and V chains two at a time.
|
||||
- Tried and not kept (slower or not exact): interleaved WMMA chains, a GELU lookup table, streaming attention, f16-accumulating WMMA, persistent or megakernel layers, phase kernels for the other deep layers, fast math.
|
||||
- **Texture kernels as native code:** `kernels/native/ptx2hip.py` translates a DLSS PTX kernel into HIP, each instruction as ZLUDA's lowering of it (`ptx_rt.h`, `native_rt.h`). `kernels/build.sh` uses it, at build time and from your own DLL, for two kernels of the fast set: `hiluma_engine_output_depthinv_mvhi_hdr_max_v2_rel` (`--ctile`, the path-B LDS tile as a function in `output_tile.h`, no unused depth prefetch, global-memory redirect stores) and `hiluma_engine_input_depthinv_mvhi_hdr_v2_rel` (`--ifconv --sink`, `in_fuse.py` moves its scratch arrays into registers, `in_regs.hip`, wave32). Both match ZLUDA's compile byte for byte in the harness. The generated sources are never stored. Other flag combinations of these kernels still use the PTX route above, and the accuracy set keeps ZLUDA's compile because the native code flushes f32 denormals.
|
||||
- **Texture kernels as native code:** `kernels/native/ptx2hip.py` translates a DLSS PTX kernel into HIP, each instruction as ZLUDA's lowering of it (`ptx_rt.h`, `native_rt.h`). `kernels/build.sh` uses it, at build time and from your own DLL, for two kernels of the fast set: `hiluma_engine_output_depthinv_mvhi_hdr_max_v2_rel` (`--ctile`, the path-B LDS tile as a function in `output_tile.h`, no unused depth prefetch, global-memory redirect stores) and `hiluma_engine_input_depthinv_mvhi_hdr_v2_rel` (`--ifconv --sink`, `in_fuse.py` moves its scratch arrays into registers, `in_regs.hip`, wave32). Both match ZLUDA's compile byte for byte in the harness. The generated sources are never stored. `cvt.rn.f16.f32` goes through `f2h_rn` (`ptx_rt.h`), whose empty asm stops LLVM from folding the conversion into the f32 operation that produced the value (`v_fma_mixlo/mixhi_f16`, which rounds the exact result to f16 once where PTX rounds to f32 and then to f16; that differed in 1 of ~29,000 random f16 operand triples on an RX 7700 XT). Before that, the output kernels left single 1-ulp values (3 history and 1 output value of a 960×540 → 1920×1080 frame) that the temporal history spread to 72 dB after 30 frames; since, every buffer the mvlo output kernel writes equals ZLUDA's compile over 12 frames, and the presented frames of all four HDR max variants are byte-identical over 30 frames (performance, quality 1129×636, ultra performance, DLAA, exposure changes, history resets, feature recreation). The input kernels compile to the same code as before. Other flag combinations of these kernels still use the PTX route above, and the accuracy set keeps ZLUDA's compile because the native code flushes f32 denormals.
|
||||
|
||||
**DLSS 4.5, presets L/M (`kernels/m`, `rrlite_*`).**
|
||||
- **Network:** the Swin blocks of DLSS 4.5, whose weights are FP8. Their prep kernels expand the weights to f16 WMMA operands. The enc3 tube layer prepares on every launch: NGX can recycle a weights address for a different tube block after feature recreation, so persistent address-only reuse can return stale weights and corrupt the image after quality changes. Other layers retain their existing preparation policy.
|
||||
@@ -263,9 +263,9 @@ kernels/build.sh all DIR # everything into DIR (default kernels/out/native)
|
||||
D4R_PREFER_ACCURACY=1 kernels/build.sh all DIR/accuracy # conservative versions of every family
|
||||
```
|
||||
|
||||
`D4R_GPU_ARCH` selects one target for this standalone build (default gfx1101). The release script builds the network layers for gfx1100–gfx1103 and gfx1200–gfx1201 by default and puts each set in its own directory, plus K's layers for the RDNA2 targets gfx1030–gfx1036. The network kernels use gfx11 or gfx12 WMMA through `kernels/common/wmma_layout.h`.
|
||||
`D4R_GPU_ARCH` selects one target for this standalone build (default gfx1101). The release script builds the network layers for gfx1100–gfx1103 and gfx1200–gfx1201 by default and puts each set in its own directory, plus K's layers and K's natively translated hiluma kernels (below) for the RDNA2 targets gfx1030–gfx1036. The network kernels use gfx11 or gfx12 WMMA through `kernels/common/wmma_layout.h`.
|
||||
|
||||
RDNA2 has no WMMA instruction. On gfx103x (or with `D4R_WMMA_SOFT=1` on an RDNA3 target, for validation) `wm_mma` keeps the gfx11 register layout and does each 16×16×16 product with 64 `v_dot2c_f32_f16` instructions whose `row_share` DPP broadcast supplies the A row, after one `row_ror` per A dword that offsets half 1 by a row. Every accumulator sums its K pairs in the WMMA order: on the RX 7700 XT the result is bit-identical to the WMMA instruction for random 64-step chains (denormals included) and for K's whole network over a moving 24-frame harness sequence. There K takes 3.6 ms per 1080p frame against 1.48 ms with WMMA and 46.8 ms fully translated without WMMA (ZLUDA's dot-product fallback, what RDNA2 ran before). M's layers and the texture tails still call WMMA directly, so RDNA2 builds only K; LLVM does not fold a `v_mov_b32_dpp` into `v_dot2c`, so the fused instruction is inline assembly. Texture-kernel builds use `D4R_ZLUDA_EMIT` to generate code objects for the selected target without that GPU. Set `D4R_NATIVE_FP8=1` on gfx12 for the matching FP8 M and texture variants.
|
||||
RDNA2 has no WMMA instruction. On gfx103x (or with `D4R_WMMA_SOFT=1` on an RDNA3 target, for validation) `wm_mma` keeps the gfx11 register layout and does each 16×16×16 product with 64 `v_dot2c_f32_f16` instructions whose `row_share` DPP broadcast supplies the A row, after one `row_ror` per A dword that offsets half 1 by a row. Every accumulator sums its K pairs in the WMMA order: on the RX 7700 XT the result is bit-identical to the WMMA instruction for random 64-step chains (denormals included) and for K's whole network over a moving 24-frame harness sequence. There K takes 3.6 ms per 1080p frame against 1.48 ms with WMMA and 46.8 ms fully translated without WMMA (ZLUDA's dot-product fallback, what RDNA2 ran before). M's layers and the texture tails still call WMMA directly, so RDNA2 builds only K's layers and the hiluma kernels that `ptx2hip.py` translates (`kernels/build.sh tex` on a gfx103x target: the four HDR max output variants and the eight input variants; `m`, `l`, `rr` and `rrswin` are refused). Those contain no matrix products. The output kernels are wave32 there (RDNA2 issues wave64 in two passes; wave64 was RDNA3's FP32 dual-issue gain) and WGP mode: at 96 VGPRs a gfx103x SIMD holds 10 waves, and a 256-thread workgroup takes 2 of them per SIMD in WGP mode instead of 4, so 10 fit instead of 8. With RDNA2's occupancy imposed on the RX 7700 XT (a register clobber to 144 VGPRs) that measured 0.596 → 0.568 ms, same bytes; capping registers to 80 for 12 waves spills and was slower (0.567 against 0.553 ms with the occupancy imposed through the LDS size). On the RX 7700 XT at 960×540 → 1920×1080 the output kernel takes 0.79 ms translated and 0.44 ms native. ZLUDA's own gfx1032 compile of it uses 131 VGPRs (7 waves); the native one 96. RDNA2 timings are not measured yet; LLVM does not fold a `v_mov_b32_dpp` into `v_dot2c`, so the fused instruction is inline assembly. Texture-kernel builds use `D4R_ZLUDA_EMIT` to generate code objects for the selected target without that GPU. Set `D4R_NATIVE_FP8=1` on gfx12 for the matching FP8 M and texture variants.
|
||||
|
||||
The release includes both fast and accuracy network sets. Full releases also include accuracy texture sets: supply them in `D4R_BUNDLE_TEX/accuracy/<target>` or set `D4R_ZLUDA_EMIT` so packaging builds them. Build accuracy sets into a separate directory; the builder refuses to mark a directory containing unmarked older binaries as accurate.
|
||||
|
||||
|
||||
+5
-2
@@ -167,14 +167,17 @@ if [[ ! -f "$BUILDER" ]]; then
|
||||
[[ "$DO_ENGINE" != 1 ]] || fail "missing $BUILDER (this release has no engine-model builder)"
|
||||
done_msg; exit 0
|
||||
fi
|
||||
models=(rr-d rr-e rr-f k k-ldr m m-ldr l l-ldr)
|
||||
models=(rr-d rr-e rr-f rr-d-gfx11 rr-e-gfx11 k k-ldr m m-ldr l l-ldr)
|
||||
# One build serves every game with the same DLLs and builder: the key covers both and the accuracy choice.
|
||||
key() { # key EXACT(0|1)
|
||||
{ echo "exact=$1"
|
||||
# Model compilers accept per-layer K_DEFS_*, M_* arithmetic/tile options, and D4R_RR_* choices.
|
||||
# A cache hit must not silently replace a model built with different compiler options.
|
||||
env | LC_ALL=C sort | sed -n -e '/^K_/p' -e '/^M_/p' -e '/^D4R_RR_/p'
|
||||
sha256sum -- "$RUNTIME/nvngx_dlss.dll" "$RUNTIME/nvngx_dlssd.dll" 2>/dev/null | cut -d' ' -f1
|
||||
(cd "$RUNTIME/tools" && find . -type f ! -name '*.pyc' -print0 | LC_ALL=C sort -z | xargs -0 sha256sum); } | sha256sum | cut -c1-16
|
||||
}
|
||||
have() { local m; for m in "${models[@]}"; do [[ -f "$1/$m/model.json" || -f "$1/$m/rr.txt" ]] || return 1; done; }
|
||||
have() { python3 "$RUNTIME/tools/models/model_tools.py" validate "$1" 2>/dev/null; }
|
||||
STAMP="$RUNTIME/engine/.d4r-models"
|
||||
if [[ "$EXACT" == ask ]]; then
|
||||
EXACT=0
|
||||
|
||||
+25
-8
@@ -5,8 +5,10 @@
|
||||
# usage: kernels/build.sh [all|k|l|m|tex|rr|rrswin] [OUT_DIR] (default: all, kernels/out/native)
|
||||
#
|
||||
# D4R_ROCM_DIR ROCm installation with clang and the HIP device libraries (default /opt/rocm)
|
||||
# D4R_GPU_ARCH target GPU (default gfx1101): an RDNA3 (gfx110x) or RDNA4 (gfx120x) target; the kernels
|
||||
# need wave32 WMMA, and kernels/common/wmma_layout.h picks the register layout per target
|
||||
# D4R_GPU_ARCH target GPU (default gfx1101): an RDNA3 (gfx110x) or RDNA4 (gfx120x) target, where
|
||||
# kernels/common/wmma_layout.h picks the WMMA register layout per target, or an RDNA2 (gfx103x)
|
||||
# target: there only K's layers (software matrix products) and the texture kernels that
|
||||
# need no WMMA (tex: K's natively translated hiluma output and input kernels) are built
|
||||
# D4R_NATIVE_FP8 1 (RDNA4 only): the variant for ZLUDA's native FP8 WMMA (d4r.ini [Kernels] NativeFp8):
|
||||
# M's layers on FP8 WMMA with NVIDIA's e4m3 activations, texture tails matching ZLUDA's
|
||||
# D4R_ZLUDA_WMMA_FP8_NATIVE lowering. K is the same in both variants.
|
||||
@@ -45,6 +47,12 @@ CLANG="$ROCM/lib/llvm/bin/clang++"
|
||||
[[ -x "$CLANG" ]] || { echo "clang++ not found in $ROCM/lib/llvm/bin (set D4R_ROCM_DIR)" >&2; exit 2; }
|
||||
mkdir -p "$OUT"
|
||||
case "$WHAT" in all|k|l|m|tex|rr|rrswin) ;; *) echo "unknown kernel family: $WHAT" >&2; exit 2 ;; esac
|
||||
# RDNA2 has no WMMA instruction: M/L's Swin layers, their texture tails and Ray Reconstruction call it directly
|
||||
RDNA2=0
|
||||
if [[ "$ARCH" == gfx103* ]]; then
|
||||
RDNA2=1
|
||||
case "$WHAT" in m|l|rr|rrswin) echo "$WHAT kernels need WMMA, which RDNA2 ($ARCH) does not have" >&2; exit 2 ;; esac
|
||||
fi
|
||||
# Never certify a directory containing older fast binaries as an accuracy set.
|
||||
if [[ "$ACCURACY" == 1 ]] && compgen -G "$OUT/*.hsaco" >/dev/null &&
|
||||
{ [[ ! -f "$OUT/d4r-accuracy.txt" ]] || [[ "$(cat "$OUT/d4r-accuracy.txt")" != 1 ]]; }; then
|
||||
@@ -82,14 +90,15 @@ if [[ "$WHAT" == all || "$WHAT" == k ]]; then
|
||||
echo "== DLSS 4 (preset K) transformer layers"
|
||||
for src in "$HERE"/k/dltss_pwin_*.hip; do build_hip "$src"; done
|
||||
fi
|
||||
if [[ "$WHAT" == all || "$WHAT" == m || "$WHAT" == l ]]; then
|
||||
if [[ "$RDNA2" == 0 && ( "$WHAT" == all || "$WHAT" == m || "$WHAT" == l ) ]]; then
|
||||
echo "== DLSS 4.5 (presets L/M) shared Swin layers"
|
||||
for src in "$HERE"/m/rrlite_*.hip; do build_hip "$src" "$([[ "$FP8" == 1 ]] && echo -DD4R_FP8_WMMA)"; done
|
||||
fi
|
||||
if [[ "$WHAT" == all || "$WHAT" == tex || "$WHAT" == l ]]; then
|
||||
echo "== texture kernels (NVIDIA PTX with native parts)"
|
||||
: "${D4R_DLSS_DLL:?set D4R_DLSS_DLL to nvngx_dlss.dll (310.7)}"
|
||||
: "${D4R_ZLUDA_EMIT:?set D4R_ZLUDA_EMIT to d4r_emit from a ZLUDA build with patches/zluda}"
|
||||
# RDNA2 builds only the PTX-to-HIP translations below, which ZLUDA does not compile
|
||||
[[ "$RDNA2" == 1 ]] || : "${D4R_ZLUDA_EMIT:?set D4R_ZLUDA_EMIT to d4r_emit from a ZLUDA build with patches/zluda}"
|
||||
PTX_DIR="${D4R_DLSS_PTX_DIR:-$HERE/extracted/ptx}"
|
||||
if [[ -z "${D4R_DLSS_PTX_DIR:-}" ]]; then
|
||||
python3 "$HERE/tools/extract_dlss_ptx.py" "$D4R_DLSS_DLL" "$PTX_DIR"
|
||||
@@ -152,7 +161,10 @@ if [[ "$WHAT" == all || "$WHAT" == tex || "$WHAT" == l ]]; then
|
||||
for range in hdr ldr; do native_ins+=("hiluma_engine_input_${depth}_${mv}_${range}_v2_rel"); done
|
||||
done
|
||||
done
|
||||
for spec in "${specs[@]}"; do
|
||||
# RDNA2: none of the ZLUDA-compiled texture builds above (their tails call WMMA, and the sust_only builds
|
||||
# are not validated there); only the natively translated hiluma kernels below
|
||||
[[ "$RDNA2" == 1 ]] && specs=()
|
||||
for spec in "${specs[@]+"${specs[@]}"}"; do
|
||||
IFS=: read -r kernel src mode <<< "$spec"
|
||||
[[ "$ACCURACY" == 0 && " ${native_outs[*]} " == *" $kernel "* ]] && continue
|
||||
# accuracy sets are wave32, except M's post and downsample kernels on RDNA4: their wave64 builds keep the
|
||||
@@ -180,8 +192,13 @@ if [[ "$WHAT" == all || "$WHAT" == tex || "$WHAT" == l ]]; then
|
||||
python3 "$NAT/ptx2hip.py" --ctile --as1 "$ptx" "$native_out" > "$NT/output_translated.hip"
|
||||
python3 "$NAT/rewrite_output.py" "$NT/output_translated.hip" > "$NT/$native_out.hip"
|
||||
depth_tile=(); [[ "$native_out" == *_mvhi_* ]] && depth_tile=(-DD4R_K_NO_DEPTH_TILE)
|
||||
D4R_ROCM_DIR="$ROCM" D4R_GPU_ARCH="$ARCH" "$NAT/build_native.sh" "$NT/$native_out.hip" "$OUT/$native_out.hsaco" \
|
||||
-DD4R_NATIVE_F2I "${depth_tile[@]}" -DD4R_AS1_STORES
|
||||
# RDNA3/4: wave64 (FP32 dual issue; 0.49 -> 0.44 ms on an RX 7700 XT, same bytes). RDNA2 issues wave64
|
||||
# in two passes, so it gets wave32 (no stack frame, 96 VGPRs), and WGP mode: a 256-thread workgroup
|
||||
# then takes 2 of each SIMD's 10 wave slots instead of 4, so 10 waves fit per SIMD instead of 8
|
||||
# (measured with RDNA2's occupancy on the RX 7700 XT: 0.596 -> 0.568 ms, same bytes).
|
||||
wave=(); [[ "$RDNA2" == 1 ]] && wave=(WAVE=32 CUMODE=0)
|
||||
env "${wave[@]}" D4R_ROCM_DIR="$ROCM" D4R_GPU_ARCH="$ARCH" "$NAT/build_native.sh" "$NT/$native_out.hip" \
|
||||
"$OUT/$native_out.hsaco" -DD4R_NATIVE_F2I "${depth_tile[@]}" -DD4R_AS1_STORES
|
||||
done
|
||||
for native_in in "${native_ins[@]}"; do
|
||||
ptx="$(ptx_of "$native_in")"; [[ -n "$ptx" ]] || { echo "no PTX defines $native_in" >&2; exit 2; }
|
||||
@@ -192,7 +209,7 @@ if [[ "$WHAT" == all || "$WHAT" == tex || "$WHAT" == l ]]; then
|
||||
done
|
||||
fi
|
||||
fi
|
||||
if [[ "$WHAT" == rr || ( "$WHAT" == all && -n "${D4R_DLSSD_DLL:-}" ) ]]; then
|
||||
if [[ "$WHAT" == rr || ( "$WHAT" == all && "$RDNA2" == 0 && -n "${D4R_DLSSD_DLL:-}" ) ]]; then
|
||||
echo "== Ray Reconstruction texture kernels"
|
||||
: "${D4R_DLSSD_DLL:?set D4R_DLSSD_DLL to the CUDA-capable nvngx_dlssd.dll}"
|
||||
: "${D4R_ZLUDA_EMIT:?set D4R_ZLUDA_EMIT to d4r_emit from a patched ZLUDA build}"
|
||||
|
||||
@@ -61,6 +61,17 @@ __device__ __forceinline__ uint32_t lane_id()
|
||||
#undef PWIN_NATIVE_F16ACC
|
||||
#undef PWIN_F16ACC_NATIVE
|
||||
#endif
|
||||
// RDNA2: the weight images use plain global loads. The raw-buffer path below builds its resource with gfx11's
|
||||
// dword3 (0x31004000: unified format 4, 8_SSCALED), which gfx11 ignores for untyped loads; gfx10 descriptors want
|
||||
// a 32-bit format there (AMD's own libraries use 0x31014000 on gfx103x), and no gfx10 encoding has been validated
|
||||
// on RDNA2 hardware, so RDNA2 takes no descriptor at all. Keyed on D4R_WMMA_SOFT (always set on gfx10) so that the
|
||||
// RDNA3 validation build of the RDNA2 kernels runs the same loads.
|
||||
#ifdef D4R_WMMA_SOFT
|
||||
#ifndef PWIN_NO_BUFFER_IMG
|
||||
#define PWIN_NO_BUFFER_IMG
|
||||
#endif
|
||||
#undef PWIN_GELU_LUT_BUFFER
|
||||
#endif
|
||||
typedef wm_op op_t; // WMMA operand: u8v (gfx11) or u4v (gfx12)
|
||||
|
||||
// one k16 step with the f16 accumulator of NVIDIA's f16 wmma (rounded after the step)
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# build_native.sh SRC.hip OUT.hsaco [extra clang flags]: natively compiled DLSS texture kernel (gfx1101, wave64 default)
|
||||
# WAVE=32|64 (default 64); CUMODE=1|0 (default 1): CU mode, or WGP mode (a workgroup's waves on all four SIMDs)
|
||||
set -euo pipefail
|
||||
ND=$(cd "$(dirname "$0")" && pwd)
|
||||
SRC=$(realpath "$1") OUT=$(realpath -m "$2"); shift 2
|
||||
@@ -8,9 +9,10 @@ R=${D4R_ROCM_DIR:-/opt/rocm}
|
||||
ARCH=${D4R_GPU_ARCH:-gfx1101}
|
||||
W=${WAVE:-64}
|
||||
WF=$([[ $W == 64 ]] && echo -mwavefrontsize64 || echo -mno-wavefrontsize64)
|
||||
CU=$([[ ${CUMODE:-1} == 1 ]] && echo -mcumode || echo -mno-cumode)
|
||||
D=$(cd "$(dirname "$SRC")" && pwd)
|
||||
T=$(mktemp -d); trap 'rm -rf "$T"' EXIT
|
||||
(cd "$T" && "$R/lib/llvm/bin/clang++" -x hip -std=c++20 --offload-arch=$ARCH --offload-device-only -O3 $WF -mcumode \
|
||||
(cd "$T" && "$R/lib/llvm/bin/clang++" -x hip -std=c++20 --offload-arch=$ARCH --offload-device-only -O3 $WF $CU \
|
||||
-ffp-contract=off -fgpu-flush-denormals-to-zero -fno-fast-math -I"$D" -I"$ND" -I"$R/include" --rocm-path="$R" \
|
||||
--rocm-device-lib-path="$R/amdgcn/bitcode" "$@" -o "$OUT" "$SRC" -save-temps=cwd --no-gpu-bundle-output)
|
||||
S=$(ls "$T"/*.s | head -1)
|
||||
|
||||
@@ -24,8 +24,9 @@ RT void output_color_pack(f4 color, float exposure, uint32_t* a, uint32_t* b)
|
||||
const float q = div_a(ln, 9.0f) + asf(0xBC4FAE29u);
|
||||
weight = rcp_a(y) * q;
|
||||
}
|
||||
*a = pack16(ashu((h1)(y * weight)), ashu((h1)((exposure * co) * weight)));
|
||||
*b = pack16(ashu((h1)((exposure * cg) * weight)), ashu((h1)color[3]));
|
||||
// f2h_rn: rounded to f32 and then to f16, as ZLUDA's compile does (no v_fma_mixlo of the product)
|
||||
*a = pack16(ashu(f2h_rn(y * weight)), ashu(f2h_rn((exposure * co) * weight)));
|
||||
*b = pack16(ashu(f2h_rn((exposure * cg) * weight)), ashu(f2h_rn(color[3])));
|
||||
}
|
||||
|
||||
RT int output_tile_coord(int relative, int lo, int hi)
|
||||
|
||||
@@ -593,10 +593,10 @@ class Translator:
|
||||
if dt == 'f16' and stp == 'f32':
|
||||
if 'rn' not in mods:
|
||||
raise Err(op)
|
||||
A(a[0], f'(h1){V(a[1], "f32")}', 'f16'); return
|
||||
A(a[0], f'f2h_rn({V(a[1], "f32")})', 'f16'); return
|
||||
if dt == 'f16x2' and stp == 'f32':
|
||||
# d = {cvt(a) in the high half, cvt(b) in the low half}
|
||||
A(a[0], f'pack16(ashu((h1){V(a[2], "f32")}), ashu((h1){V(a[1], "f32")}))', 'u32'); return
|
||||
A(a[0], f'pack16(ashu(f2h_rn({V(a[2], "f32")})), ashu(f2h_rn({V(a[1], "f32")})))', 'u32'); return
|
||||
if dt == 'f16' and stp == 'f16':
|
||||
x = V(a[1], 'f16')
|
||||
if 'rni' in mods:
|
||||
|
||||
@@ -77,6 +77,16 @@ RT uint32_t f2u_rz(float x)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- f16 (denormals kept)
|
||||
// cvt.rn.f16.f32: the f32 value rounded to f16. The empty asm keeps LLVM from folding the conversion into the
|
||||
// f32 operation that produced the value (fptrunc(fma(fpext ...)) -> v_fma_mixlo/mixhi_f16), which rounds the exact
|
||||
// result to f16 once instead of to f32 and then to f16: on an RX 7700 XT that differed from the two-step result for
|
||||
// 1 in ~29,000 random f16 operand triples, and in the hiluma output kernel it left single 1-ulp seeds that the
|
||||
// temporal history then spread (ZLUDA keeps both roundings).
|
||||
RT h1 f2h_rn(float x)
|
||||
{
|
||||
__asm__ volatile("" : "+v"(x));
|
||||
return (h1)x;
|
||||
}
|
||||
RT h1 hfma(h1 a, h1 b, h1 c) { return __builtin_fmaf16(a, b, c); }
|
||||
RT h2 h2fma(h2 a, h2 b, h2 c) { return __builtin_elementwise_fma(a, b, c); }
|
||||
RT h1 hmaxn(h1 a, h1 b) { return __builtin_fmaximum_numf16(a, b); }
|
||||
|
||||
@@ -70,6 +70,10 @@ that archive's d4r/README.txt: setup runs automatically when the ASI loads.
|
||||
@installer compatibility tools (or set D4R_PROTON=/path/to/GE-Proton11-3). It takes a minute or two once;
|
||||
@installer the models are kept in ~/.cache/d4r/engine-models for the next game. bash install.sh --help lists
|
||||
@installer options for scripted installs.
|
||||
@installer RR D/E also have separate gfx11 models; devices without compatible matrix support retain CUDA
|
||||
@installer translation. Builds report progress and time out after 900 seconds per command (override with
|
||||
@installer D4R_MODEL_TIMEOUT). Failed captures and logs are retained under ~/.cache/d4r/model-builds;
|
||||
@installer D4R_MODEL_LOG_DIR changes that location. Model checksums are verified before cache reuse.
|
||||
@installer 3. In Steam, open the game's Properties:
|
||||
@clean 3. In Steam, open the game's Properties:
|
||||
@full 2. In Steam, open the game's Properties:
|
||||
|
||||
@@ -61,8 +61,27 @@ export PATH="$KIT/bin:$PATH" # the bundled glslangValidator
|
||||
glslangValidator --version > /dev/null 2>&1 || fail "glslangValidator ($KIT/bin) does not run on this system"
|
||||
sha() { sha256sum -- "$1" | cut -d' ' -f1; }
|
||||
|
||||
WORK="$(mktemp -d "${TMPDIR:-/tmp}/d4r-models.XXXXXX")"
|
||||
trap 'rm -rf -- "$WORK"' EXIT
|
||||
LOG_ROOT="${D4R_MODEL_LOG_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/d4r/model-builds}"
|
||||
mkdir -p -- "$LOG_ROOT"
|
||||
BUILD_LOGS="$(mktemp -d "$LOG_ROOT/build.XXXXXX")"
|
||||
WORK="$BUILD_LOGS/work"; mkdir -p "$WORK"
|
||||
MODEL_TOOLS="$KIT/model_tools.py"
|
||||
[[ -f "$MODEL_TOOLS" ]] || fail "missing $MODEL_TOOLS; update the model builder tools"
|
||||
cleanup() {
|
||||
local status=$?
|
||||
if (( status == 0 )); then
|
||||
while IFS= read -r -d '' log; do
|
||||
relative="${log#"$WORK/"}"
|
||||
mkdir -p -- "$BUILD_LOGS/$(dirname -- "$relative")"
|
||||
cp -- "$log" "$BUILD_LOGS/$relative"
|
||||
done < <(find "$WORK" -name '*.log' -type f -print0)
|
||||
rm -rf -- "$WORK"
|
||||
else
|
||||
printf 'build-engine-models: failed build and captures preserved at %s\n' "$WORK" >&2
|
||||
fi
|
||||
printf 'Model build diagnostics: %s\n' "$BUILD_LOGS"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM
|
||||
|
||||
@@ -72,6 +91,7 @@ place() { # place NAME DIR FILE...
|
||||
shift 2
|
||||
mkdir -p "$RUNTIME/engine" "$WORK/place/$name"
|
||||
for f in "$@"; do cp -- "$src/$f" "$WORK/place/$name/"; done
|
||||
python3 "$MODEL_TOOLS" manifest "$WORK/place/$name"
|
||||
rm -rf -- "$dst.new"; mv -- "$WORK/place/$name" "$dst.new"
|
||||
rm -rf -- "$dst.old"; [[ ! -e "$dst" ]] || mv -- "$dst" "$dst.old"
|
||||
mv -- "$dst.new" "$dst"; rm -rf -- "$dst.old"
|
||||
@@ -83,18 +103,24 @@ model_files() { # rr.txt and the files it names, for model directory $1
|
||||
}
|
||||
run() { # run LOG COMMAND...: the command's output in WORK/LOG, shown when it fails
|
||||
local log="$WORK/$1"; shift
|
||||
"$@" > "$log" 2>&1 || { tail -n 20 "$log" >&2; fail "failed: $(basename -- "$2") (log above)"; }
|
||||
python3 "$MODEL_TOOLS" run --log "$log" --timeout "${D4R_MODEL_TIMEOUT:-900}" -- "$@" ||
|
||||
fail "failed: $1 (see $log)"
|
||||
}
|
||||
|
||||
if (( DO_RR )); then
|
||||
DLL="$RUNTIME/nvngx_dlssd.dll"
|
||||
[[ -f "$DLL" && "$(sha "$DLL")" == "$DLSSD_SHA256" ]] || fail "Ray Reconstruction models need nvngx_dlssd.dll 310.9.1 in $RUNTIME (run install.sh)"
|
||||
echo 'Building the Ray Reconstruction models (presets D, E and F) from nvngx_dlssd.dll...'
|
||||
# Build both portable arithmetic variants; runtime selects by device capabilities. Do not let the build
|
||||
# host (or an inherited developer switch) silently determine what another game/GPU receives from the cache.
|
||||
for p in D E; do
|
||||
l=$(tr DE de <<< "$p")
|
||||
bash "$KIT/engine/rr/build_native_model.sh" --dll "$DLL" "$p" "$WORK/rr-$l" > "$WORK/rr-$l.log" 2>&1 ||
|
||||
{ tail -n 20 "$WORK/rr-$l.log" >&2; fail "preset $p model failed"; }
|
||||
place "rr-$l" "$WORK/rr-$l" $(model_files "$WORK/rr-$l") exposure_rr0.spv exposure_rr1.spv
|
||||
for arithmetic in fp8 gfx11; do
|
||||
l=$(tr DE de <<< "$p"); name="rr-$l"; gfx11=0
|
||||
if [[ "$arithmetic" == gfx11 ]]; then name+="-gfx11"; gfx11=1; fi
|
||||
run "$name.log" env D4R_RR_GFX11="$gfx11" bash "$KIT/engine/rr/build_native_model.sh" \
|
||||
--dll "$DLL" "$p" "$WORK/$name"
|
||||
place "$name" "$WORK/$name" $(model_files "$WORK/$name") exposure_rr0.spv exposure_rr1.spv
|
||||
done
|
||||
done
|
||||
run ptx-f.log python3 "$KIT/kernels/tools/extract_dlss_ptx.py" "$DLL" "$WORK/ptx-f"
|
||||
run rr-f.log python3 "$KIT/engine/compile_rr2.py" --dll "$DLL" "$WORK/rr-f" --ptx-dir "$WORK/ptx-f"
|
||||
@@ -125,16 +151,17 @@ if (( DO_SR )); then
|
||||
local w="$WORK/$1"
|
||||
mkdir -p "$w/replay"
|
||||
printf ' rendering one frame with model %s through d4r (GE-Proton %s)...\n' "$1" "$(basename "$PROTON")"
|
||||
(cd "$APP" && env STEAM_COMPAT_CLIENT_INSTALL_PATH="$STEAM_ROOT" STEAM_COMPAT_DATA_PATH="$WORK/prefix" \
|
||||
(cd "$APP" && python3 "$MODEL_TOOLS" run --log "$w/run.log" --timeout "${D4R_MODEL_TIMEOUT:-900}" -- \
|
||||
env STEAM_COMPAT_CLIENT_INSTALL_PATH="$STEAM_ROOT" STEAM_COMPAT_DATA_PATH="$WORK/prefix" \
|
||||
PROTON_FORCE_NVAPI=1 DXVK_NVAPI_GPU_ARCH=AD100 WINEDLLOVERRIDES='d3d12=n,b;d3d12core=n,b;d4r12core=n,b' \
|
||||
D4R_ENGINE=0 D4R_DLSS_PRESET="$2" D4R_HARNESS_CREATE_FLAGS=0x4B D4R_SHIM_LOG="$(winpath "$w/shim.log")" \
|
||||
D4R_CUDA_TRACE="$(winpath "$w/cuda.log")" D4R_CUDA_VERBOSE=1 \
|
||||
D4R_CUDA_REPLAY_DUMP_DIR="$w/replay" D4R_CUDA_REPLAY_DUMP_LIMIT=60 \
|
||||
"$PROTON/proton" run "$APP/d3d12_dlss_harness.exe" "$(winpath "$RUNTIME/nvngx.dll")" "$(winpath "$w/out.raw")" \
|
||||
1 640 360 1280 720) > "$w/run.log" 2>&1 || true
|
||||
if ! ls "$w/replay" | grep -q "$3"; then
|
||||
tail -n 15 "$w/run.log" >&2
|
||||
1 640 360 1280 720) || fail "model $1: capture process failed; diagnostics in $w"
|
||||
if ! python3 "$MODEL_TOOLS" capture "$w" "$1"; then
|
||||
[[ ! -f "$w/shim.log" ]] || tail -n 15 "$w/shim.log" >&2
|
||||
fail "model $1: the test frame did not run through DLSS (logs above)"
|
||||
fail "model $1: incomplete or invalid test frame (diagnostics in $w)"
|
||||
fi
|
||||
}
|
||||
echo 'Building the Super Resolution models (K, M and L) from what NGX uploads...'
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bounded model-build commands and integrity checks (also shipped in d4r/tools)."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
MANIFEST = 'files.sha256.json'
|
||||
MODELS = ('rr-d', 'rr-e', 'rr-f', 'rr-d-gfx11', 'rr-e-gfx11', 'k', 'k-ldr', 'm', 'm-ldr', 'l', 'l-ldr')
|
||||
|
||||
|
||||
def digest(path):
|
||||
with path.open('rb') as stream:
|
||||
return hashlib.file_digest(stream, 'sha256').hexdigest()
|
||||
|
||||
|
||||
def manifest(directory, write=False):
|
||||
directory = Path(directory)
|
||||
files = {str(p.relative_to(directory)): digest(p) for p in sorted(directory.rglob('*'))
|
||||
if p.is_file() and p.name != MANIFEST}
|
||||
if not files or not any(n in files for n in ('model.json', 'rr.txt')):
|
||||
raise ValueError(f'{directory}: missing model metadata')
|
||||
if write:
|
||||
(directory / MANIFEST).write_text(json.dumps(files, indent=2, sort_keys=True) + '\n')
|
||||
elif files != json.loads((directory / MANIFEST).read_text()):
|
||||
raise ValueError(f'{directory}: missing, changed or unexpected model files; rebuild the models')
|
||||
|
||||
|
||||
def capture(directory, name):
|
||||
"""A completed, finite frame and all files referenced by each network-layer capture."""
|
||||
import numpy as np
|
||||
directory = Path(directory)
|
||||
output = directory / 'out.raw'
|
||||
if output.stat().st_size != 1280 * 720 * 8:
|
||||
raise ValueError('capture did not produce a complete RGBA16F frame')
|
||||
pixels = np.fromfile(output, dtype=np.float16).reshape(-1, 4)
|
||||
if not np.isfinite(pixels).all() or not np.any(pixels[:, :3]):
|
||||
raise ValueError('capture produced a nonfinite or empty frame')
|
||||
if name == 'k':
|
||||
kernels = ['dltss_pwin_' + s + '_layer' for s in
|
||||
('enc0', 'enc1', 'enc2', 'enc3', 'enc4', 'dec5', 'dec4', 'dec3', 'dec2', 'dec1', 'dec0')]
|
||||
else:
|
||||
kernels = ['rrlite_' + s + '_4x4' for s in ('enc1', 'enc2', 'enc3', 'enc4', 'enc5', 'dec4', 'dec3', 'dec2', 'dec1')]
|
||||
kernels += ['rrlite_enc0_4x4_mvlo_hdr' + ('_folded' if name == 'm' else ''),
|
||||
'rrlite_dec0_4x4' + ('_folded' if name == 'm' else '')]
|
||||
for kernel in kernels:
|
||||
matches = sorted((directory / 'replay').glob('replay-*-' + kernel))
|
||||
if not matches:
|
||||
raise ValueError(f'missing captured layer {kernel}')
|
||||
launch = matches[0]
|
||||
rows = [line.split() for line in (launch / 'manifest.txt').read_text().splitlines()]
|
||||
if ['kernel', kernel] not in rows:
|
||||
raise ValueError(f'{kernel}: incorrect capture metadata')
|
||||
args = next(int(r[1]) for r in rows if r[0] == 'args')
|
||||
if args <= 0 or (launch / 'args.bin').stat().st_size != args:
|
||||
raise ValueError(f'{kernel}: truncated arguments')
|
||||
allocations = {r[1]: int(r[3]) for r in rows if r[0] == 'alloc'}
|
||||
if not allocations:
|
||||
raise ValueError(f'{kernel}: no weight allocations captured')
|
||||
for index, size in allocations.items():
|
||||
if size <= 0 or (launch / f'alloc-{index}.bin').stat().st_size != size:
|
||||
raise ValueError(f'{kernel}: truncated allocation {index}')
|
||||
for r in rows:
|
||||
if r[0] == 'pointer' and (r[2] not in allocations or not 0 <= int(r[3]) < allocations[r[2]]):
|
||||
raise ValueError(f'{kernel}: invalid captured pointer')
|
||||
|
||||
|
||||
def run(log, timeout, command):
|
||||
if command and command[0] == '--':
|
||||
command = command[1:]
|
||||
if not command or not 0 < timeout <= 86400:
|
||||
raise ValueError('command required; timeout must be between 0 and 86400 seconds')
|
||||
log = Path(log)
|
||||
log.parent.mkdir(parents=True, exist_ok=True)
|
||||
started = time.monotonic()
|
||||
print(f' starting {log.stem} (deadline {timeout:g}s; log {log})', flush=True)
|
||||
with log.open('wb') as output:
|
||||
child = subprocess.Popen(command, stdout=output, stderr=subprocess.STDOUT, start_new_session=True)
|
||||
def stop():
|
||||
for sig in (signal.SIGTERM, signal.SIGKILL):
|
||||
try:
|
||||
os.killpg(child.pid, sig)
|
||||
except ProcessLookupError:
|
||||
break
|
||||
try:
|
||||
child.wait(timeout=3)
|
||||
except subprocess.TimeoutExpired:
|
||||
continue
|
||||
# Kill remaining members too, even when the group leader has exited.
|
||||
if sig == signal.SIGTERM:
|
||||
try:
|
||||
os.killpg(child.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
break
|
||||
previous = {}
|
||||
def interrupted(sig, frame):
|
||||
raise KeyboardInterrupt
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
previous[sig] = signal.signal(sig, interrupted)
|
||||
result = None
|
||||
try:
|
||||
heartbeat = 0
|
||||
while child.poll() is None:
|
||||
elapsed = time.monotonic() - started
|
||||
if elapsed >= timeout:
|
||||
stop()
|
||||
print(f' {log.stem}: timed out after {elapsed:.0f}s', file=sys.stderr)
|
||||
result = 124
|
||||
break
|
||||
if elapsed >= heartbeat + 15:
|
||||
print(f' {log.stem}: still running ({elapsed:.0f}s, pid {child.pid}); log {log}', flush=True)
|
||||
heartbeat = elapsed
|
||||
time.sleep(min(.25, max(.01, timeout - elapsed)))
|
||||
if result is None:
|
||||
result = child.returncode
|
||||
except KeyboardInterrupt:
|
||||
stop()
|
||||
result = 130
|
||||
finally:
|
||||
for sig, handler in previous.items():
|
||||
signal.signal(sig, handler)
|
||||
if result:
|
||||
print('\n'.join(log.read_text(errors='replace').splitlines()[-30:]), file=sys.stderr)
|
||||
print(f' {log.stem}: exit={result}, elapsed={time.monotonic() - started:.1f}s; log {log}', flush=True)
|
||||
return result if result >= 0 else 128 - result
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest='action', required=True)
|
||||
for action in ('manifest', 'validate'):
|
||||
sub.add_parser(action).add_argument('directory', type=Path)
|
||||
p = sub.add_parser('capture')
|
||||
p.add_argument('directory', type=Path)
|
||||
p.add_argument('name', choices=('k', 'm', 'l'))
|
||||
p = sub.add_parser('run')
|
||||
p.add_argument('--log', required=True)
|
||||
p.add_argument('--timeout', type=float, default=900)
|
||||
p.add_argument('command', nargs=argparse.REMAINDER)
|
||||
a = parser.parse_args()
|
||||
try:
|
||||
if a.action == 'run':
|
||||
return run(a.log, a.timeout, a.command)
|
||||
if a.action == 'capture':
|
||||
capture(a.directory, a.name)
|
||||
elif a.action == 'manifest':
|
||||
manifest(a.directory, write=True)
|
||||
else:
|
||||
for name in MODELS:
|
||||
manifest(a.directory / name)
|
||||
except (OSError, ValueError, KeyError, StopIteration) as error:
|
||||
print(f'model-tools: {error}', file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -6,6 +6,7 @@ set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
MINGW_CXX="${MINGW_CXX:-x86_64-w64-mingw32-g++}"
|
||||
CLANG_CL="${CLANG_CL:-clang-cl}"
|
||||
BUILD_ID="$(bash "$ROOT/scripts/runtime_build_id.sh")"
|
||||
|
||||
VULKAN_INCLUDE="${VULKAN_INCLUDE:-/usr/include}"
|
||||
|
||||
@@ -22,7 +23,7 @@ PARAM_OBJECT="$ROOT/build/d4r_ngx_param_msvc.obj"
|
||||
"$CLANG_CL" --target=x86_64-pc-windows-msvc /nologo /std:c++20 /O2 /c /GS- /GR- /EHs-c- /Zl \
|
||||
"/Fo$PARAM_OBJECT" -- "$ROOT/tools/d4r_ngx_param_msvc.cpp"
|
||||
# The engine (D4R_ENGINE) reaches Vulkan through a function table: MinGW has no Vulkan import library.
|
||||
"$MINGW_CXX" -std=c++20 -O2 -Wall -Wextra -Wno-missing-field-initializers -DD4R_ENGINE_VK_TABLE -I"$ROOT/build/vulkan-include" -I"$ROOT/build/vulkan-staging" -shared -static -static-libgcc -static-libstdc++ \
|
||||
"$MINGW_CXX" -std=c++20 -O2 -Wall -Wextra -Wno-missing-field-initializers -DD4R_ENGINE_VK_TABLE "-DD4R_RUNTIME_BUILD_ID=\"$BUILD_ID\"" -I"$ROOT/build/vulkan-include" -I"$ROOT/build/vulkan-staging" -shared -static -static-libgcc -static-libstdc++ \
|
||||
"$ROOT/tools/d4r_nvngx_shim.cpp" "$ROOT/tools/d4r_ngx_param_host.cpp" "$PARAM_OBJECT" \
|
||||
"$ROOT/engine/runtime.cpp" "$ROOT/engine/game.cpp" "$ROOT/engine/game_m.cpp" "$ROOT/engine/game_rr.cpp" "$ROOT/engine/vk_dispatch.cpp" \
|
||||
-ldxgi -lbcrypt -Wl,--entry,d4r_dll_entry -o "$ROOT/build/d4r_nvngx.dll"
|
||||
|
||||
@@ -76,8 +76,14 @@ export D4R_DLSS_PTX_DIR="$BUILD/ptx"
|
||||
python3 "$ROOT/kernels/tools/extract_dlss_ptx.py" "$D4R_BUNDLE_DLSS" "$D4R_DLSS_PTX_DIR" > "$BUILD/logs/extract.log"
|
||||
build_textures() {
|
||||
local arch="$1" folder fp8 accuracy out
|
||||
# RDNA2 has no texture tails (they need WMMA); package_release.sh builds its K layers
|
||||
[[ "$arch" == gfx103* ]] && return 0
|
||||
# RDNA2: only K's natively translated hiluma output/input kernels (no WMMA), fast set only (the accuracy
|
||||
# set keeps ZLUDA's compile of them); package_release.sh builds the K layers and copies these
|
||||
if [[ "$arch" == gfx103* ]]; then
|
||||
D4R_PREFER_ACCURACY=0 D4R_GPU_ARCH="$arch" D4R_DLSS_DLL="$D4R_BUNDLE_DLSS" \
|
||||
"$ROOT/kernels/build.sh" tex "$D4R_BUNDLE_TEX/$arch" > "$BUILD/logs/tex-$arch-0.log" 2>&1
|
||||
echo "Texture kernels rebuilt: $arch (native hiluma output/input only)"
|
||||
return 0
|
||||
fi
|
||||
folders=("$arch")
|
||||
[[ "$arch" == gfx12* ]] && folders+=("$arch-fp8")
|
||||
for folder in "${folders[@]}"; do
|
||||
|
||||
@@ -5,6 +5,7 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
WINEGCC="${WINEGCC:-winegcc}"
|
||||
WINEBUILD="${WINEBUILD:-winebuild}"
|
||||
BRIDGE_ROOT="$ROOT/build/wine-nvcuda"
|
||||
BUILD_ID="$(bash "$ROOT/scripts/runtime_build_id.sh")"
|
||||
|
||||
mkdir -p "$BRIDGE_ROOT/x86_64-unix" "$BRIDGE_ROOT/x86_64-windows"
|
||||
MOTION_FLAGS=()
|
||||
@@ -13,6 +14,7 @@ if [[ -x "${D4R_ROCM_DIR:-/opt/rocm}/lib/llvm/bin/clang++" ]]; then
|
||||
MOTION_FLAGS=(-DD4R_MOTION_KERNELS -I"$BRIDGE_ROOT/generated")
|
||||
fi
|
||||
"$WINEGCC" -m64 -shared -o "$BRIDGE_ROOT/x86_64-unix/nvcuda.dll" \
|
||||
"-DD4R_RUNTIME_BUILD_ID=\"$BUILD_ID\"" \
|
||||
"${MOTION_FLAGS[@]}" \
|
||||
"$ROOT/tools/wine_nvcuda_bridge.c" \
|
||||
-Wb,--export="$ROOT/tools/wine_nvcuda_bridge.spec" -ldl -lpthread
|
||||
|
||||
@@ -206,6 +206,12 @@ def main(argv):
|
||||
env["D4R_SHIM_MARKER_POLL_US"] = value
|
||||
|
||||
value = flag("Engine", "Enabled")
|
||||
if value is None:
|
||||
models = Path(os.environ.get("D4R_RUNTIME_DIR", path.parent)) / "engine"
|
||||
value = any((models / model).is_dir() for model in
|
||||
("k", "k-ldr", "m", "m-ldr", "l", "l-ldr", "rr-d", "rr-e", "rr-f",
|
||||
"rr-d-gfx11", "rr-e-gfx11"))
|
||||
env["D4R_ENGINE_K"] = "0"
|
||||
env["D4R_ENGINE"] = "1" if value else "0"
|
||||
if path_value("Engine", "ModelDir"):
|
||||
env["D4R_ENGINE_MODEL_DIR"] = path_value("Engine", "ModelDir")
|
||||
|
||||
@@ -16,7 +16,8 @@
|
||||
# D4R_ROCM_RUNTIME ROCm runtime bundled as d4r/rocm (scripts/fetch_rocm_runtime.sh)
|
||||
# default ~/.cache/d4r-rocm-runtime
|
||||
# D4R_MAX_GLIBC highest GLIBC ABI the Linux libraries may require default 2.41
|
||||
# D4R_GPU_ARCHS GPU targets to build kernels for default gfx1030-gfx1036 (K only), gfx1100-gfx1103, gfx1200, gfx1201
|
||||
# D4R_GPU_ARCHS GPU targets to build kernels for default gfx1030-gfx1036 (K and its hiluma kernels only),
|
||||
# gfx1100-gfx1103, gfx1200, gfx1201
|
||||
# (RDNA4 targets also get a <target>-fp8 folder: the kernels for d4r.ini NativeFp8 = on)
|
||||
# D4R_OPTISCALER_LICENSE OptiScaler's LICENSE (GPL-3.0) text; default: system SPDX or common-licenses copy
|
||||
# D4R_VKD3D_SRC vkd3d-proton source checkout, for its license files default ~/.cache/d4r-vkd3d-proton
|
||||
@@ -216,15 +217,30 @@ ensure_rr_textures() {
|
||||
fi
|
||||
}
|
||||
build_target() {
|
||||
local folder="$1" arch fp8 tex_dir accurate accurate_tex f
|
||||
local folder="$1" arch fp8 tex_dir tmp_tex accurate accurate_tex f
|
||||
arch="${folder%-fp8}"
|
||||
fp8=0
|
||||
[[ "$folder" == *-fp8 ]] && fp8=1
|
||||
# RDNA2 (gfx103x): only DLSS 4's layers (preset K) have native kernels there, built with software matrix
|
||||
# products (kernels/common/wmma_layout.h); M, L, the texture tails and Ray Reconstruction stay translated by ZLUDA
|
||||
# RDNA2 (gfx103x): DLSS 4's layers (preset K), built with software matrix products
|
||||
# (kernels/common/wmma_layout.h), and K's hiluma output/input kernels translated from the PTX
|
||||
# (kernels/native, no WMMA; wave32, byte-identical to ZLUDA's compile). M, L, the texture tails and Ray
|
||||
# Reconstruction call WMMA and stay translated by ZLUDA. The accuracy set keeps ZLUDA's compile of hiluma.
|
||||
if [[ "$arch" == gfx103* ]]; then
|
||||
D4R_PREFER_ACCURACY=0 D4R_ROCM_DIR="$ROCM" D4R_GPU_ARCH="$arch" "$ROOT/kernels/build.sh" k "$STAGE/d4r/kernels/$folder" >/dev/null
|
||||
D4R_PREFER_ACCURACY=1 D4R_ROCM_DIR="$ROCM" D4R_GPU_ARCH="$arch" "$ROOT/kernels/build.sh" k "$STAGE/d4r/kernels/accuracy/$folder" >/dev/null
|
||||
if [[ "$VARIANT" != clean ]]; then
|
||||
tex_dir="$D4R_BUNDLE_TEX/$folder" tmp_tex=""
|
||||
if [[ ! -d "$tex_dir" ]]; then
|
||||
tex_dir="$(mktemp -d)" tmp_tex="$tex_dir"
|
||||
D4R_PREFER_ACCURACY=0 D4R_ROCM_DIR="$ROCM" D4R_GPU_ARCH="$arch" D4R_DLSS_DLL="$D4R_BUNDLE_DLSS" \
|
||||
"$ROOT/kernels/build.sh" tex "$tex_dir" >/dev/null
|
||||
fi
|
||||
# only the kernels validated for RDNA2, whatever else an older bundle folder holds
|
||||
for f in "$tex_dir"/hiluma_engine_output_*_hdr_max_v2_rel.hsaco "$tex_dir"/hiluma_engine_input_*_v2_rel.hsaco; do
|
||||
[[ -f "$f" ]] && cp "$f" "$STAGE/d4r/kernels/$folder/"
|
||||
done
|
||||
[[ -z "$tmp_tex" ]] || rm -rf "$tmp_tex"
|
||||
fi
|
||||
rm -f "$STAGE/d4r/kernels/$folder"/*.resolution.txt "$STAGE/d4r/kernels/accuracy/$folder"/*.resolution.txt
|
||||
python3 "$ROOT/kernels/tools/kernel_manifest.py" "$STAGE/d4r/kernels/$folder" "${DLLS[@]}" "${RR_DLLS[@]}"
|
||||
python3 "$ROOT/kernels/tools/kernel_manifest.py" "$STAGE/d4r/kernels/accuracy/$folder" "${DLLS[@]}" "${RR_DLLS[@]}"
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
|
||||
revision=$(git rev-parse --short HEAD 2>/dev/null || echo unknown)
|
||||
sources=$(sha256sum tools/*.h tools/d4r_nvngx_shim.cpp tools/d4r_vulkan_frontend.inl tools/wine_nvcuda_bridge.c \
|
||||
engine/*.h engine/*.cpp engine/*.inc | sha256sum | cut -c1-16)
|
||||
printf '%s-%s\n' "$revision" "$sources"
|
||||
@@ -47,5 +47,6 @@ cp "$GL/bin/glslangValidator" "$KIT/bin/glslangValidator"
|
||||
chmod +x "$KIT/bin/glslangValidator"
|
||||
cp "$D4R_GLSLANG_LICENSE" "$D4R/licenses/glslang-LICENSE.txt"
|
||||
cp "$ROOT/packaging/build_engine_models.sh" "$D4R/tools/build-engine-models.sh"
|
||||
cp "$ROOT/packaging/model_tools.py" "$KIT/model_tools.py"
|
||||
chmod +x "$D4R/tools/build-engine-models.sh"
|
||||
printf 'Staged the engine-model builder in %s (%s)\n' "$D4R/tools" "$(du -sh "$D4R/tools" | cut -f1)"
|
||||
@@ -0,0 +1,149 @@
|
||||
"""GPU identity and model capability decisions without a GPU or Wine process."""
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
class GpuCompatibilityTests(unittest.TestCase):
|
||||
def compile_run(self, source, cxx=False):
|
||||
with tempfile.TemporaryDirectory() as t:
|
||||
executable = Path(t) / 'check'
|
||||
result = subprocess.run(['c++' if cxx else 'cc', '-x', 'c++' if cxx else 'c',
|
||||
'-std=c++17' if cxx else '-std=gnu11', '-Wall', '-Wextra', '-Werror',
|
||||
'-pthread',
|
||||
'-I', str(ROOT / 'tools'), '-o', str(executable), '-'], input=source,
|
||||
capture_output=True, text=True)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
subprocess.run([str(executable)], check=True, timeout=5)
|
||||
|
||||
def test_pci_identity_is_not_a_vendor_id_or_gpu_size_guess(self):
|
||||
self.compile_run(r'''
|
||||
#include "d4r_gpu_identity.h"
|
||||
#include <assert.h>
|
||||
int main(void) {
|
||||
assert(d4r_same_pci("0000:03:00.0", "00000000:03:00.0"));
|
||||
assert(!d4r_same_pci("0000:03:00.0", "0000:04:00.0"));
|
||||
assert(!d4r_same_pci("0000:03:00.0", "0001:03:00.0"));
|
||||
assert(!d4r_same_pci("0000:03:00.0garbage", "0000:03:00.0"));
|
||||
assert(d4r_pci_matches_kfd("0000:03:00.0", 0, 0x300));
|
||||
assert(!d4r_pci_matches_kfd("0000:03:00.0", 0, 0x400));
|
||||
assert(!d4r_pci_matches_kfd("0000:03:00.0", 1, 0x300));
|
||||
}
|
||||
''')
|
||||
|
||||
def test_bridge_maps_nonzero_compute_ordinal_and_only_its_luid(self):
|
||||
source = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
|
||||
selection = source[source.index('static CUresult selected_cuda_device('):
|
||||
source.index('static void create_default_zluda_context(')]
|
||||
luid = source[source.index('CUresult WINAPI cuDeviceGetLuid('):
|
||||
source.index('CUresult WINAPI cuDeviceGetAttribute(')]
|
||||
self.compile_run(r'''
|
||||
#include "d4r_gpu_identity.h"
|
||||
#include <assert.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#define WINAPI
|
||||
typedef int CUresult;
|
||||
typedef int CUdevice;
|
||||
typedef int (*CUDEVICEGETCOUNT_FN)(int*);
|
||||
typedef int (*CUDEVICEGET_FN)(int*, int);
|
||||
typedef int (*CUDEVICEGETLUID_FN)(char*, unsigned*, int);
|
||||
enum { CUDA_SUCCESS=0, CUDA_ERROR_NOT_INITIALIZED=3, CUDA_ERROR_NOT_SUPPORTED=801 };
|
||||
static const char* render = "0000:03:00.0";
|
||||
static int devices = 2;
|
||||
static unsigned GetEnvironmentVariableA(const char* key, char* out, unsigned size) {
|
||||
(void)key; snprintf(out, size, "%s", render); return (unsigned)strlen(render);
|
||||
}
|
||||
static void tracef(const char* fmt, ...) { (void)fmt; }
|
||||
static int count(int* n) { *n=devices; return 0; }
|
||||
static int get(int* d,int n) { *d=100+n; return 0; }
|
||||
static int pci(char* out,int size,int device) {
|
||||
snprintf(out, (size_t)size, "%s", device==101 ? "00000000:03:00.0" : "0000:04:00.0"); return 0;
|
||||
}
|
||||
static int original_luid(char* out,unsigned* mask,int device) { (void)device; memset(out,0,8);*mask=0;return 0; }
|
||||
static void* find_zluda_symbol(const char* name) {
|
||||
if (!strcmp(name,"cuDeviceGetCount")) return (void*)count;
|
||||
if (!strcmp(name,"cuDeviceGet")) return (void*)get;
|
||||
if (!strcmp(name,"cuDeviceGetPCIBusId")) return (void*)pci;
|
||||
return (void*)original_luid;
|
||||
}
|
||||
static int missing(const char* name) { (void)name; return 999; }
|
||||
static unsigned get_process_u32(const char* name,unsigned fallback) {
|
||||
(void)fallback;
|
||||
return strstr(name,"LOW") ? 0x3f2 : strstr(name,"HIGH") ? 0 : 1;
|
||||
}
|
||||
''' + selection + luid + r'''
|
||||
int main(void) {
|
||||
assert(d4r_pci_matches_kfd("0000:03:00.0",0,0x300));
|
||||
int selected=-1; assert(selected_cuda_device(&selected)==0 && selected==101);
|
||||
char id[8]; unsigned mask=9;
|
||||
assert(cuDeviceGetLuid(id,&mask,100)==0 && mask==0);
|
||||
assert(cuDeviceGetLuid(id,&mask,101)==0 && mask==1);
|
||||
unsigned low=0;memcpy(&low,id,4);assert(low==0x3f2);
|
||||
render="0000:05:00.0";assert(selected_cuda_device(&selected)==801);
|
||||
render="";assert(selected_cuda_device(&selected)==801);
|
||||
devices=1;assert(selected_cuda_device(&selected)==0 && selected==100);
|
||||
}
|
||||
''')
|
||||
|
||||
def test_rr_model_cannot_reach_unsupported_fp8_pipelines(self):
|
||||
self.compile_run(r'''
|
||||
#include "d4r_engine_compat.h"
|
||||
#include <cassert>
|
||||
int main() {
|
||||
const char* fp8 = "D4RR0001\npreset 4\nstage enc0 x.spv\n";
|
||||
const char* wide = "D4RR0001\npreset 4\narithmetic gfx11\nstage enc0 x.spv\n";
|
||||
assert(!d4r_rr_model_compatible(fp8, true, false));
|
||||
assert(!d4r_rr_model_compatible(wide, false, false));
|
||||
assert(d4r_rr_model_compatible(wide, true, false));
|
||||
assert(d4r_rr_model_compatible(fp8, true, true));
|
||||
assert(!d4r_rr_model_compatible("D4RR0002\npreset 6\n", true, false));
|
||||
assert(!d4r_rr_model_compatible("D4RR0001\narithmetic unknown\n", true, true));
|
||||
VkCooperativeMatrixPropertiesKHR p{};
|
||||
p.MSize=p.NSize=p.KSize=16; p.scope=VK_SCOPE_SUBGROUP_KHR;
|
||||
p.AType=p.BType=VK_COMPONENT_TYPE_FLOAT16_KHR;
|
||||
p.CType=p.ResultType=VK_COMPONENT_TYPE_FLOAT32_KHR;
|
||||
assert(d4r_matrix_tuple(p, VK_COMPONENT_TYPE_FLOAT16_KHR));
|
||||
p.KSize=32; assert(!d4r_matrix_tuple(p, VK_COMPONENT_TYPE_FLOAT16_KHR));
|
||||
}
|
||||
''', cxx=True)
|
||||
|
||||
def test_existing_context_is_preserved_and_bootstrap_binds_each_thread(self):
|
||||
text = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
|
||||
ensure = text[text.index('static void ensure_context(void)'):text.index('/* D4R_NGX_STREAM_PRIORITY')]
|
||||
self.compile_run(r'''
|
||||
#include <assert.h>
|
||||
#include <pthread.h>
|
||||
#include <stdint.h>
|
||||
#include <string.h>
|
||||
typedef int CUresult;
|
||||
typedef void* CUcontext;
|
||||
typedef CUresult (*CUCTX_CURRENT_FN)(CUcontext);
|
||||
enum { CUDA_SUCCESS=0 };
|
||||
static pthread_once_t context_once=PTHREAD_ONCE_INIT;
|
||||
static int context_bootstrap_result=3, bootstrap_calls=0;
|
||||
static _Thread_local int context_setup_result=3;
|
||||
static _Thread_local void* active;
|
||||
static void* cuda_context;
|
||||
static int current(void** out) { *out=active;return 0; }
|
||||
static int set(void* context) { active=context;return 0; }
|
||||
static void* find_zluda_symbol(const char* key) {
|
||||
return !strcmp(key,"cuCtxGetCurrent") ? (void*)current : (void*)set;
|
||||
}
|
||||
static void create_default_zluda_context(void) {
|
||||
++bootstrap_calls;cuda_context=(void*)(uintptr_t)0x456;context_bootstrap_result=0;
|
||||
}
|
||||
''' + ensure + r'''
|
||||
static void* worker(void* ignored) {
|
||||
(void)ignored;ensure_context();assert(active==cuda_context && context_setup_result==0);return NULL;
|
||||
}
|
||||
int main(void) {
|
||||
active=(void*)(uintptr_t)0x123;ensure_context();assert(active==(void*)(uintptr_t)0x123 && bootstrap_calls==0);
|
||||
pthread_t a;assert(!pthread_create(&a,NULL,worker,NULL));pthread_join(a,NULL);
|
||||
assert(bootstrap_calls==1);ensure_context();assert(active==(void*)(uintptr_t)0x123);
|
||||
assert(!pthread_create(&a,NULL,worker,NULL));pthread_join(a,NULL);assert(bootstrap_calls==1);
|
||||
}
|
||||
''')
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Model builds must time out, retain failures, and reject incomplete/corrupt artifacts."""
|
||||
import contextlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
spec = importlib.util.spec_from_file_location('model_tools', ROOT / 'packaging/model_tools.py')
|
||||
model_tools = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(model_tools)
|
||||
|
||||
|
||||
class ModelBuildTests(unittest.TestCase):
|
||||
def test_installer_key_tracks_model_arithmetic_options(self):
|
||||
text = (ROOT / 'install.sh').read_text()
|
||||
key = text[text.index('key() {'):text.index('have() {')]
|
||||
with tempfile.TemporaryDirectory() as t:
|
||||
root = Path(t)
|
||||
(root / 'tools').mkdir()
|
||||
(root / 'tools/compiler.py').write_text('compiler fixture')
|
||||
for name in ('nvngx_dlss.dll', 'nvngx_dlssd.dll'): (root / name).write_bytes(b'DLL fixture')
|
||||
env = {k: v for k, v in os.environ.items() if not k.startswith(('D4R_RR_', 'K_', 'M_'))}
|
||||
command = ['bash', '-c', 'RUNTIME=$1\n' + key + '\nkey 0', '_', t]
|
||||
baseline = subprocess.check_output(command, env=env)
|
||||
for name in ('K_DEFS_ENC0', 'M_EXACT', 'D4R_RR_MIX'):
|
||||
self.assertNotEqual(baseline, subprocess.check_output(command, env=dict(env, **{name: '1'})))
|
||||
|
||||
def test_child_exit_status_and_timeout_are_preserved(self):
|
||||
with tempfile.TemporaryDirectory() as t, contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
|
||||
log = Path(t) / 'capture.log'
|
||||
self.assertEqual(model_tools.run(log, 5, [sys.executable, '-c', 'print("failure detail");raise SystemExit(37)']), 37)
|
||||
self.assertIn('failure detail', log.read_text())
|
||||
self.assertEqual(model_tools.run(log, .15, [sys.executable, '-c', 'import time;time.sleep(60)']), 124)
|
||||
self.assertTrue(log.exists())
|
||||
|
||||
def test_manifest_rejects_missing_modified_and_extra_files(self):
|
||||
with tempfile.TemporaryDirectory() as t:
|
||||
directory = Path(t)
|
||||
(directory / 'model.json').write_text('{}')
|
||||
weights = directory / 'weights.bin'
|
||||
weights.write_bytes(b'weights')
|
||||
model_tools.manifest(directory, write=True)
|
||||
model_tools.manifest(directory)
|
||||
weights.write_bytes(b'corrupt')
|
||||
with self.assertRaises(ValueError): model_tools.manifest(directory)
|
||||
weights.unlink()
|
||||
with self.assertRaises(ValueError): model_tools.manifest(directory)
|
||||
weights.write_bytes(b'weights')
|
||||
(directory / 'unexpected.spv').write_bytes(b'extra')
|
||||
with self.assertRaises(ValueError): model_tools.manifest(directory)
|
||||
|
||||
def test_capture_requires_completed_frame_and_all_layers(self):
|
||||
import numpy as np
|
||||
with tempfile.TemporaryDirectory() as t:
|
||||
directory = Path(t)
|
||||
pixels = np.ones((720, 1280, 4), dtype=np.float16)
|
||||
pixels.tofile(directory / 'out.raw')
|
||||
replay = directory / 'replay'
|
||||
replay.mkdir()
|
||||
for layer in ('enc0', 'enc1', 'enc2', 'enc3', 'enc4', 'dec5', 'dec4', 'dec3', 'dec2', 'dec1', 'dec0'):
|
||||
kernel = 'dltss_pwin_' + layer + '_layer'
|
||||
d = replay / ('replay-000001-' + kernel)
|
||||
d.mkdir()
|
||||
(d / 'manifest.txt').write_text(f'kernel {kernel}\nargs 8\nalloc 0 0x1000 4\npointer 0 0 0\n')
|
||||
(d / 'args.bin').write_bytes(bytes(8))
|
||||
(d / 'alloc-0.bin').write_bytes(bytes(4))
|
||||
model_tools.capture(directory, 'k')
|
||||
(d / 'alloc-0.bin').write_bytes(b'')
|
||||
with self.assertRaises(ValueError): model_tools.capture(directory, 'k')
|
||||
(d / 'alloc-0.bin').write_bytes(bytes(4))
|
||||
pixels[0, 0, 0] = np.nan
|
||||
pixels.tofile(directory / 'out.raw')
|
||||
with self.assertRaises(ValueError): model_tools.capture(directory, 'k')
|
||||
|
||||
def test_dump_does_not_accept_a_failed_child_with_a_replay_marker(self):
|
||||
text = (ROOT / 'packaging/build_engine_models.sh').read_text()
|
||||
function = text[text.index(' dump() {'):text.index(" echo 'Building the Super Resolution")]
|
||||
with tempfile.TemporaryDirectory() as t:
|
||||
root = Path(t)
|
||||
(root / 'proton').mkdir(); (root / 'app').mkdir()
|
||||
proton = root / 'proton/proton'
|
||||
proton.write_text('#!/bin/sh\nmkdir -p "$D4R_CUDA_REPLAY_DUMP_DIR/replay-000001-dltss_pwin_dec0_layer"\nexit 37\n')
|
||||
proton.chmod(0o755)
|
||||
prefix = 'set -euo pipefail\nWORK=$1; APP=$WORK/app; PROTON=$WORK/proton; STEAM_ROOT=$WORK; RUNTIME=$WORK\n'
|
||||
prefix += 'MODEL_TOOLS=$2; winpath() { printf "%s" "$1"; }; fail() { echo "$*" >&2; exit 1; }\n'
|
||||
result = subprocess.run(['bash', '-c', prefix + function + '\ndump k 11 ignored\n', '_', t,
|
||||
str(ROOT / 'packaging/model_tools.py')], capture_output=True, text=True, timeout=10)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn('capture process failed', result.stderr)
|
||||
self.assertTrue((root / 'k/run.log').exists())
|
||||
@@ -0,0 +1,193 @@
|
||||
"""RDNA2 (gfx103x) native hiluma kernels: build gating, packaging allowlist and the f16 rounding contract.
|
||||
|
||||
Run: python3 -m unittest tests.test_rdna2_native_output -v
|
||||
The compile checks need ROCm's clang (D4R_ROCM_DIR, default /opt/rocm) and are skipped without it. The full
|
||||
texture build additionally needs the extracted DLSS 310.7 PTX (kernels/extracted/ptx) and
|
||||
D4R_RUN_KERNEL_BUILD_TESTS=1, since it compiles all twelve kernels.
|
||||
Arithmetic equality with ZLUDA's compile is checked in the harness (docs/native-kernels.md), not here.
|
||||
"""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ROCM = Path(os.environ.get("D4R_ROCM_DIR", "/opt/rocm"))
|
||||
CLANG = ROCM / "lib/llvm/bin/clang++"
|
||||
PTX_DIR = ROOT / "kernels/extracted/ptx"
|
||||
TOOLS = [shutil.which(t) for t in ("llvm-readelf", "llvm-objdump")]
|
||||
OUTPUTS = {f"hiluma_engine_output_{d}_{mv}_hdr_max_v2_rel" for d in ("depthinv", "depthreg") for mv in ("mvhi", "mvlo")}
|
||||
INPUTS = {f"hiluma_engine_input_{d}_{mv}_{r}_v2_rel"
|
||||
for d in ("depthinv", "depthreg") for mv in ("mvhi", "mvlo") for r in ("hdr", "ldr")}
|
||||
# cvt.rn.f16.f32 of an fma (and the packed form of a product) of widened f16 values: LLVM would fold the
|
||||
# conversion into v_fma_mixlo/mixhi_f16, which rounds once instead of twice
|
||||
TINY_PTX = """.version 8.0
|
||||
.target sm_80
|
||||
.address_size 64
|
||||
.visible .entry tiny(
|
||||
.param .align 8 .b8 tiny_param_0[16]
|
||||
)
|
||||
{
|
||||
.reg .b16 %rs<6>;
|
||||
.reg .f32 %f<8>;
|
||||
.reg .b32 %r<3>;
|
||||
.reg .b64 %rd<3>;
|
||||
ld.param.u64 %rd1, [tiny_param_0];
|
||||
cvta.to.global.u64 %rd2, %rd1;
|
||||
ld.global.u16 %rs2, [%rd2+8];
|
||||
ld.global.u16 %rs3, [%rd2+10];
|
||||
ld.global.u16 %rs4, [%rd2+12];
|
||||
cvt.f32.f16 %f3, %rs2;
|
||||
cvt.f32.f16 %f4, %rs3;
|
||||
cvt.f32.f16 %f5, %rs4;
|
||||
fma.rn.f32 %f2, %f3, %f4, %f5;
|
||||
cvt.rn.f16.f32 %rs1, %f2;
|
||||
mul.rn.f32 %f6, %f3, %f4;
|
||||
cvt.rn.f16x2.f32 %r1, %f6, %f2;
|
||||
st.global.u16 [%rd2], %rs1;
|
||||
st.global.u32 [%rd2+4], %r1;
|
||||
ret;
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def run(*args, **kwargs):
|
||||
return subprocess.run([str(a) for a in args], text=True, capture_output=True, **kwargs)
|
||||
|
||||
|
||||
class Rdna2BuildGating(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
# a compiler that fails if anything reaches it: the refusals must come first
|
||||
compiler = self.root / "rocm/lib/llvm/bin/clang++"
|
||||
compiler.parent.mkdir(parents=True)
|
||||
compiler.write_text("#!/bin/sh\nexit 99\n")
|
||||
compiler.chmod(0o755)
|
||||
|
||||
def test_wmma_families_are_refused_on_every_rdna2_target(self):
|
||||
for arch in ("gfx1030", "gfx1032", "gfx1036"):
|
||||
for family in ("m", "l", "rr", "rrswin"):
|
||||
env = dict(os.environ, D4R_ROCM_DIR=str(self.root / "rocm"), D4R_GPU_ARCH=arch)
|
||||
result = run("bash", ROOT / "kernels/build.sh", family, self.root / "out", env=env)
|
||||
self.assertEqual(result.returncode, 2, (arch, family, result.stderr))
|
||||
self.assertIn("need WMMA", result.stderr)
|
||||
|
||||
def test_rdna2_texture_build_does_not_need_zluda(self):
|
||||
env = dict(os.environ, D4R_ROCM_DIR=str(self.root / "rocm"), D4R_GPU_ARCH="gfx1032",
|
||||
D4R_DLSS_DLL=str(self.root / "missing.dll"), D4R_DLSS_PTX_DIR=str(self.root / "no-ptx"))
|
||||
env.pop("D4R_ZLUDA_EMIT", None)
|
||||
result = run("bash", ROOT / "kernels/build.sh", "tex", self.root / "out", env=env)
|
||||
self.assertNotIn("D4R_ZLUDA_EMIT", result.stderr)
|
||||
self.assertIn("missing extracted PTX directory", result.stderr)
|
||||
|
||||
def test_rdna3_texture_build_still_needs_zluda(self):
|
||||
env = dict(os.environ, D4R_ROCM_DIR=str(self.root / "rocm"), D4R_GPU_ARCH="gfx1101",
|
||||
D4R_DLSS_DLL=str(self.root / "missing.dll"))
|
||||
env.pop("D4R_ZLUDA_EMIT", None)
|
||||
result = run("bash", ROOT / "kernels/build.sh", "tex", self.root / "out", env=env)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("D4R_ZLUDA_EMIT", result.stderr)
|
||||
|
||||
|
||||
class Rdna2Packaging(unittest.TestCase):
|
||||
def rdna2_branch(self, path):
|
||||
text = (ROOT / path).read_text()
|
||||
start = text.index('if [[ "$arch" == gfx103* ]]; then')
|
||||
return text[start:text.index("\n fi\n", start)]
|
||||
|
||||
def test_package_copies_only_the_validated_hiluma_kernels(self):
|
||||
branch = self.rdna2_branch("scripts/package_release.sh")
|
||||
copies = re.findall(r'"\$tex_dir"/([^ ;]+\.hsaco)', branch)
|
||||
self.assertEqual(sorted(copies), ["hiluma_engine_input_*_v2_rel.hsaco",
|
||||
"hiluma_engine_output_*_hdr_max_v2_rel.hsaco"])
|
||||
self.assertIn('"$VARIANT" != clean', branch) # the no-NVIDIA zip ships no PTX-derived code
|
||||
self.assertNotIn("build.sh\" m", branch)
|
||||
# the accuracy set gets K's layers only (ZLUDA keeps compiling hiluma there)
|
||||
self.assertNotRegex(branch, r'PREFER_ACCURACY=1[^\n]*\n?[^\n]*build.sh" tex')
|
||||
self.assertIn("kernel_manifest.py", branch)
|
||||
|
||||
def test_release_builder_builds_rdna2_fast_textures_only(self):
|
||||
text = (ROOT / "scripts/build_release_glibc241.sh").read_text()
|
||||
start = text.index('if [[ "$arch" == gfx103* ]]; then')
|
||||
branch = text[start:text.index("fi\n", start)]
|
||||
self.assertIn("D4R_PREFER_ACCURACY=0", branch)
|
||||
self.assertNotIn("D4R_PREFER_ACCURACY=1", branch)
|
||||
self.assertIn('kernels/build.sh" tex', branch)
|
||||
self.assertIn("return 0", branch)
|
||||
|
||||
|
||||
class F16RoundingContract(unittest.TestCase):
|
||||
def translate(self, directory):
|
||||
ptx = directory / "tiny.ptx"
|
||||
ptx.write_text(TINY_PTX)
|
||||
result = run("python3", ROOT / "kernels/native/ptx2hip.py", ptx, "tiny", check=True)
|
||||
return result.stdout
|
||||
|
||||
def test_translator_routes_f16_conversions_through_f2h_rn(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
source = self.translate(Path(temp))
|
||||
self.assertEqual(source.count("f2h_rn("), 3)
|
||||
self.assertNotRegex(source, r"\(h1\)f\d")
|
||||
|
||||
@unittest.skipUnless(CLANG.is_file(), "ROCm clang not found (set D4R_ROCM_DIR)")
|
||||
def test_no_single_rounding_mix_instructions(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
temp = Path(temp)
|
||||
(temp / "tiny.hip").write_text(self.translate(temp))
|
||||
for arch, wave in (("gfx1032", "32"), ("gfx1101", "64")):
|
||||
env = dict(os.environ, D4R_ROCM_DIR=str(ROCM), D4R_GPU_ARCH=arch, WAVE=wave)
|
||||
run("bash", ROOT / "kernels/native/build_native.sh", temp / "tiny.hip", temp / f"{arch}.hsaco",
|
||||
env=env, check=True)
|
||||
asm = (temp / f"{arch}.s").read_text()
|
||||
self.assertNotRegex(asm, r"v_fma_mix(lo|hi)_f16", arch)
|
||||
# canary: a plain cast in the same kernel is fused, so the check above can fail
|
||||
(temp / "cast.hip").write_text((temp / "tiny.hip").read_text().replace("f2h_rn(", "(h1)("))
|
||||
env = dict(os.environ, D4R_ROCM_DIR=str(ROCM), D4R_GPU_ARCH="gfx1032", WAVE="32")
|
||||
run("bash", ROOT / "kernels/native/build_native.sh", temp / "cast.hip", temp / "cast.hsaco",
|
||||
env=env, check=True)
|
||||
self.assertRegex((temp / "cast.s").read_text(), r"v_fma_mix(lo|hi)_f16")
|
||||
|
||||
|
||||
@unittest.skipUnless(os.environ.get("D4R_RUN_KERNEL_BUILD_TESTS") == "1", "set D4R_RUN_KERNEL_BUILD_TESTS=1")
|
||||
@unittest.skipUnless(CLANG.is_file() and PTX_DIR.is_dir() and all(TOOLS), "needs ROCm clang, LLVM tools and PTX")
|
||||
class Rdna2TextureBuild(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.temp = tempfile.TemporaryDirectory()
|
||||
cls.out = Path(cls.temp.name) / "gfx1032"
|
||||
env = dict(os.environ, D4R_ROCM_DIR=str(ROCM), D4R_GPU_ARCH="gfx1032", D4R_DLSS_DLL="unused",
|
||||
D4R_DLSS_PTX_DIR=str(PTX_DIR))
|
||||
run("bash", ROOT / "kernels/build.sh", "tex", cls.out, env=env, check=True)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls.temp.cleanup()
|
||||
|
||||
def notes(self, name):
|
||||
return run(TOOLS[0], "--notes", "-h", self.out / f"{name}.hsaco", check=True).stdout
|
||||
|
||||
def test_exactly_the_validated_kernels(self):
|
||||
self.assertEqual({p.stem for p in self.out.glob("*.hsaco")}, OUTPUTS | INPUTS)
|
||||
|
||||
def test_code_objects_are_wave32_gfx1032_without_scratch_or_wmma(self):
|
||||
for name in OUTPUTS | INPUTS:
|
||||
notes = self.notes(name)
|
||||
self.assertIn("gfx1032", notes, name)
|
||||
self.assertRegex(notes, r"\.wavefront_size:\s+32", name)
|
||||
self.assertRegex(notes, r"\.private_segment_fixed_size:\s+0\b", name)
|
||||
code = run(TOOLS[1], "-d", "--mcpu=gfx1032", self.out / f"{name}.hsaco", check=True).stdout
|
||||
self.assertNotRegex(code, r"v_wmma|v_fma_mix(lo|hi)_f16|v_dot\d", name)
|
||||
|
||||
def test_output_kernels_use_wgp_mode(self):
|
||||
for name in OUTPUTS:
|
||||
asm = (self.out / f"{name}.s").read_text()
|
||||
self.assertIn(".amdhsa_workgroup_processor_mode 1", asm, name)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -60,6 +60,20 @@ class ShimConfigTests(unittest.TestCase):
|
||||
result = subprocess.run(cmd, env=env, text=True, capture_output=True)
|
||||
self.assertEqual(result.returncode, 2)
|
||||
|
||||
def test_auto_engine_detects_models_without_overriding_environment(self):
|
||||
with tempfile.TemporaryDirectory() as t:
|
||||
root = Path(t)
|
||||
ini = root / 'd4r.ini'
|
||||
ini.write_text('[Engine]\nEnabled = auto\n')
|
||||
(root / 'engine/rr-d-gfx11').mkdir(parents=True)
|
||||
env = {k: v for k, v in os.environ.items() if not k.startswith('D4R_')}
|
||||
cmd = ['python3', str(ROOT / 'scripts/d4r_config.py'), '--config', str(ini)]
|
||||
output = subprocess.check_output(cmd, env=env, text=True)
|
||||
self.assertIn('export D4R_ENGINE=1\n', output)
|
||||
self.assertIn('export D4R_ENGINE_K=0\n', output)
|
||||
output = subprocess.check_output(cmd, env=dict(env, D4R_ENGINE='0'), text=True)
|
||||
self.assertNotIn('export D4R_ENGINE=', output)
|
||||
|
||||
def test_launcher_marker_poll_defaults_overrides_and_validation(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
ini = Path(temporary) / 'd4r.ini'
|
||||
|
||||
@@ -127,14 +127,29 @@ static bool check(HRESULT hr, const char* what)
|
||||
|
||||
static void submit_and_wait()
|
||||
{
|
||||
g_list->Close();
|
||||
if (!check(g_list->Close(), "Close command list")) std::_Exit(1);
|
||||
ID3D12CommandList* lists[] = {g_list};
|
||||
g_queue->ExecuteCommandLists(1, lists);
|
||||
g_queue->Signal(g_fence, ++g_fenceValue);
|
||||
if (!check(g_queue->Signal(g_fence, ++g_fenceValue), "Signal fence")) std::_Exit(1);
|
||||
if (g_fence->GetCompletedValue() < g_fenceValue)
|
||||
{
|
||||
g_fence->SetEventOnCompletion(g_fenceValue, g_event);
|
||||
WaitForSingleObject(g_event, INFINITE);
|
||||
if (!check(g_fence->SetEventOnCompletion(g_fenceValue, g_event), "SetEventOnCompletion")) std::_Exit(1);
|
||||
DWORD timeout = 60000;
|
||||
if (const char* value = std::getenv("D4R_HARNESS_GPU_TIMEOUT_MS"))
|
||||
{
|
||||
char* end = nullptr;
|
||||
const unsigned long parsed = std::strtoul(value, &end, 10);
|
||||
if (end != value && *end == '\0' && parsed >= 1 && parsed <= 600000) timeout = parsed;
|
||||
}
|
||||
const DWORD waited = WaitForSingleObject(g_event, timeout);
|
||||
if (waited != WAIT_OBJECT_0)
|
||||
{
|
||||
std::fprintf(stderr, "GPU fence wait failed: wait=0x%lx timeout=%lu ms fence=%llu completed=%llu device=0x%08lx\n",
|
||||
waited, timeout, g_fenceValue, g_fence->GetCompletedValue(), g_device->GetDeviceRemovedReason());
|
||||
std::fflush(stderr);
|
||||
// The capture is an isolated diagnostic process. Do not enter teardown with outstanding GPU work.
|
||||
std::_Exit(waited == WAIT_TIMEOUT ? 124 : 1);
|
||||
}
|
||||
}
|
||||
g_allocator->Reset();
|
||||
g_list->Reset(g_allocator, nullptr);
|
||||
@@ -1043,6 +1058,9 @@ static float halton(uint32_t index, uint32_t base)
|
||||
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
// The installer redirects output to a file. Keep the last stage visible even if HIP/Wine stalls or crashes.
|
||||
std::setvbuf(stdout, nullptr, _IONBF, 0);
|
||||
std::setvbuf(stderr, nullptr, _IONBF, 0);
|
||||
trace_stage("entered main");
|
||||
if (argc < 3)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
#pragma once
|
||||
#ifndef D4R_RUNTIME_BUILD_ID
|
||||
#define D4R_RUNTIME_BUILD_ID "unidentified (built without runtime_build_id.sh)"
|
||||
#endif
|
||||
@@ -0,0 +1,35 @@
|
||||
#pragma once
|
||||
#include <vulkan/vulkan_core.h>
|
||||
#include <sstream>
|
||||
#include <string>
|
||||
|
||||
// Inspect metadata before constructing any pipeline. Old default D/E and F models require FP8;
|
||||
// only the explicitly marked gfx11 D/E models use widened f16 cooperative matrices.
|
||||
inline bool d4r_rr_model_compatible(const std::string& metadata, bool f16Matrices, bool fp8Matrices)
|
||||
{
|
||||
std::istringstream lines(metadata);
|
||||
std::string line;
|
||||
if (!std::getline(lines, line)) return false;
|
||||
if (!line.empty() && line.back() == '\r') line.pop_back();
|
||||
if (line == "D4RR0002") return fp8Matrices;
|
||||
if (line != "D4RR0001") return false;
|
||||
bool widened = false;
|
||||
while (std::getline(lines, line))
|
||||
{
|
||||
std::istringstream fields(line);
|
||||
std::string key, value; fields >> key >> value;
|
||||
if (key == "arithmetic")
|
||||
{
|
||||
if (value != "gfx11") return false;
|
||||
widened = true;
|
||||
}
|
||||
}
|
||||
return widened ? f16Matrices : fp8Matrices;
|
||||
}
|
||||
|
||||
inline bool d4r_matrix_tuple(const VkCooperativeMatrixPropertiesKHR& p, VkComponentTypeKHR operand)
|
||||
{
|
||||
return p.MSize == 16 && p.NSize == 16 && p.KSize == 16 && p.scope == VK_SCOPE_SUBGROUP_KHR &&
|
||||
p.AType == operand && p.BType == operand && p.CType == VK_COMPONENT_TYPE_FLOAT32_KHR &&
|
||||
p.ResultType == VK_COMPONENT_TYPE_FLOAT32_KHR && !p.saturatingAccumulation;
|
||||
}
|
||||
@@ -73,7 +73,7 @@ inline bool d4r_read_hex_file(const char* path, uint32_t& value)
|
||||
|
||||
// The real PCI identity of the GPU behind a vkd3d-proton device: Vulkan's PCI bus address, then that
|
||||
// device's sysfs vendor/device files.
|
||||
inline D4rPciIdentity d4r_true_identity(ID3D12Device* device, std::string& detail)
|
||||
inline D4rPciIdentity d4r_true_identity(ID3D12Device* device, std::string& detail, std::string* address = nullptr)
|
||||
{
|
||||
D4rPciIdentity identity;
|
||||
ID3D12DXVKInteropDevice1* interop = nullptr;
|
||||
@@ -126,6 +126,12 @@ inline D4rPciIdentity d4r_true_identity(ID3D12Device* device, std::string& detai
|
||||
return identity;
|
||||
}
|
||||
identity.known = true;
|
||||
if (address != nullptr)
|
||||
{
|
||||
char pci[32];
|
||||
std::snprintf(pci, sizeof(pci), "%04x:%02x:%02x.%x", bus.pciDomain, bus.pciBus, bus.pciDevice, bus.pciFunction);
|
||||
*address = pci;
|
||||
}
|
||||
char text[256];
|
||||
std::snprintf(text, sizeof(text), "PCI %04x:%02x:%02x.%x is %04x:%04x (Vulkan reports %04x:%04x, driver %s)",
|
||||
bus.pciDomain, bus.pciBus, bus.pciDevice, bus.pciFunction, identity.vendor, identity.device,
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
/* PCI identity comparisons shared by the Wine bridge and CPU tests. */
|
||||
#pragma once
|
||||
#include <stdio.h>
|
||||
|
||||
typedef struct D4rPciAddress { unsigned domain, bus, device, function; } D4rPciAddress;
|
||||
|
||||
static int d4r_parse_pci(const char* text, D4rPciAddress* out)
|
||||
{
|
||||
char extra;
|
||||
return text != NULL && sscanf(text, "%x:%x:%x.%x%c", &out->domain, &out->bus,
|
||||
&out->device, &out->function, &extra) == 4 && out->bus <= 255 && out->device <= 31 && out->function <= 7;
|
||||
}
|
||||
|
||||
static int d4r_same_pci(const char* a, const char* b)
|
||||
{
|
||||
D4rPciAddress x, y;
|
||||
return d4r_parse_pci(a, &x) && d4r_parse_pci(b, &y) && x.domain == y.domain &&
|
||||
x.bus == y.bus && x.device == y.device && x.function == y.function;
|
||||
}
|
||||
|
||||
static int d4r_pci_matches_kfd(const char* pci, unsigned long domain, unsigned long location)
|
||||
{
|
||||
D4rPciAddress address;
|
||||
return d4r_parse_pci(pci, &address) && address.domain == domain &&
|
||||
((address.bus << 8) | (address.device << 3) | address.function) == location;
|
||||
}
|
||||
@@ -101,6 +101,8 @@
|
||||
#include "d4r_fsr_rr.h"
|
||||
#include "d4r_fsr_rr_identity.h"
|
||||
#include "d4r_fsr4_identity.h"
|
||||
#include "d4r_engine_compat.h"
|
||||
#include "d4r_build.h"
|
||||
#include "d4r_fsr_rr_passes.h"
|
||||
#include "../engine/game.h"
|
||||
#include "../engine/game_m.h"
|
||||
@@ -752,7 +754,7 @@ static void load_portable_config()
|
||||
bool on = false;
|
||||
if (enabled.empty() || enabled == "auto")
|
||||
{
|
||||
for (const wchar_t* model : {L"k", L"k-ldr", L"m", L"m-ldr", L"l", L"l-ldr", L"rr-d", L"rr-e", L"rr-f"})
|
||||
for (const wchar_t* model : {L"k", L"k-ldr", L"m", L"m-ldr", L"l", L"l-ldr", L"rr-d", L"rr-e", L"rr-f", L"rr-d-gfx11", L"rr-e-gfx11"})
|
||||
{
|
||||
const DWORD attributes = GetFileAttributesW((dir + L"\\engine\\" + model).c_str());
|
||||
if (attributes != INVALID_FILE_ATTRIBUTES && (attributes & FILE_ATTRIBUTE_DIRECTORY))
|
||||
@@ -956,12 +958,16 @@ public:
|
||||
return jobs_.size();
|
||||
}
|
||||
|
||||
template <typename Function> auto call(Function function) -> decltype(function())
|
||||
template <typename Function> auto call(Function function, const char* label = nullptr) -> decltype(function())
|
||||
{
|
||||
using Result = decltype(function());
|
||||
auto task = std::make_shared<std::packaged_task<Result()>>(std::move(function));
|
||||
auto future = task->get_future();
|
||||
post([task] { (*task)(); });
|
||||
const auto started = std::chrono::steady_clock::now();
|
||||
while (label && future.wait_for(std::chrono::seconds(15)) == std::future_status::timeout)
|
||||
logf("worker wait: %s still pending after %.0f s (resources retained)", label,
|
||||
std::chrono::duration<double>(std::chrono::steady_clock::now() - started).count());
|
||||
return future.get();
|
||||
}
|
||||
|
||||
@@ -1974,6 +1980,7 @@ static NgxResult initialize(unsigned long long applicationId, const wchar_t* dat
|
||||
ensure_portable_config();
|
||||
if (g_portable.active && !g.core)
|
||||
{
|
||||
logf("d4r runtime build %s", D4R_RUNTIME_BUILD_ID);
|
||||
logf("portable install in %ls (settings from d4r.ini, Linux path %s)", g_portable.dir.c_str(),
|
||||
g_portable.unixDir.c_str());
|
||||
for (const std::string& note : g_portable.notes)
|
||||
@@ -1993,6 +2000,11 @@ static NgxResult initialize(unsigned long long applicationId, const wchar_t* dat
|
||||
LUID luid;
|
||||
device->GetAdapterLuid(&luid);
|
||||
set_cuda_adapter_luid(luid);
|
||||
std::string adapterDetail, adapterPci;
|
||||
d4r_true_identity(device, adapterDetail, &adapterPci);
|
||||
SetEnvironmentVariableA("D4R_RENDER_PCI", adapterPci.empty() ? nullptr : adapterPci.c_str());
|
||||
logf("CUDA adapter mapping: %s; PCI match %s", adapterDetail.c_str(),
|
||||
adapterPci.empty() ? "unavailable (multiple compute devices will be refused)" : adapterPci.c_str());
|
||||
// OptiScaler's FSR 4 option needs the real AMD identity for OptiScaler and AMD's modules (d4r_fsr4_identity.h)
|
||||
if (env_string("D4R_FSR4_IDENTITY") != "0")
|
||||
{
|
||||
@@ -2003,6 +2015,8 @@ static NgxResult initialize(unsigned long long applicationId, const wchar_t* dat
|
||||
}
|
||||
|
||||
g.paths.clear();
|
||||
logf("engine selection: D4R_ENGINE=%s D4R_ENGINE_K=%s network=%s (environment overrides d4r.ini)",
|
||||
env_string("D4R_ENGINE").c_str(), env_string("D4R_ENGINE_K").c_str(), env_string("D4R_ENGINE_NETWORK").c_str());
|
||||
wchar_t featureDirectory[MAX_PATH] = {};
|
||||
DWORD length = GetEnvironmentVariableW(L"D4R_NGX_FEATURE_DIR", featureDirectory, MAX_PATH);
|
||||
if (length > 0 && length < MAX_PATH)
|
||||
@@ -3404,6 +3418,7 @@ struct VulkanInterop
|
||||
// shaderBufferInt64Atomics is enabled on vkd3d-proton's device (it enables it when the GPU has it): Ray
|
||||
// Reconstruction preset F then measures the exposure in one pass, as NGX does, instead of two
|
||||
bool int64Atomics = false;
|
||||
bool matrixF16 = false;
|
||||
// The engine runs on the game's own Vulkan device (native Vulkan NGX frontend): no vkd3d interop objects.
|
||||
bool native = false;
|
||||
PFN_vkGetPhysicalDeviceFormatProperties formatProperties = nullptr;
|
||||
@@ -8952,6 +8967,9 @@ static int start_engine_model_rr(const Feature& feature);
|
||||
// Vulkan-only K features retain just their creation settings until a frame actually needs CUDA.
|
||||
static NgxResult create_cuda_feature(Feature& feature)
|
||||
{
|
||||
const auto createStarted = std::chrono::steady_clock::now();
|
||||
logf("CUDA_CREATE_BEGIN preset=%u render=%ux%u output=%ux%u flags=0x%x", feature.preset,
|
||||
feature.width, feature.height, feature.outWidth, feature.outHeight, feature.flags);
|
||||
static const char* const kSuperSamplingPresets[] = {
|
||||
"DLSS.Hint.Render.Preset.DLAA", "DLSS.Hint.Render.Preset.Quality", "DLSS.Hint.Render.Preset.Balanced",
|
||||
"DLSS.Hint.Render.Preset.Performance", "DLSS.Hint.Render.Preset.UltraPerformance",
|
||||
@@ -8963,6 +8981,7 @@ static NgxResult create_cuda_feature(Feature& feature)
|
||||
const bool rayReconstruction = feature.rayReconstruction;
|
||||
const char* const* presetNames = rayReconstruction ? kRayReconstructionPresets : kSuperSamplingPresets;
|
||||
const NgxResult result = g.worker.call([&]() -> NgxResult {
|
||||
logf("CUDA_CREATE_WORKER_BEGIN preset=%u", feature.preset);
|
||||
NgxResult allocation = g.ngx.allocateParameters(&feature.cudaParams);
|
||||
if (allocation != NGX_SUCCESS)
|
||||
return allocation;
|
||||
@@ -9002,8 +9021,10 @@ static NgxResult create_cuda_feature(Feature& feature)
|
||||
feature.scratch = 0;
|
||||
// The core owns the denoiser: it loads nvngx_dlssd.dll, knows its kernel map, and
|
||||
// dispatches feature 13 to it. Only the feature id differs from Super Sampling.
|
||||
logf("CUDA_CREATE_NGX_BEGIN preset=%u scratch=0x%llx", feature.preset, feature.scratch);
|
||||
const NgxResult created = g.ngx.createFeature(
|
||||
rayReconstruction ? NGX_FEATURE_RAY_RECONSTRUCTION : NGX_FEATURE_SUPER_SAMPLING, p, &feature.cudaHandle);
|
||||
logf("CUDA_CREATE_NGX_END preset=%u result=0x%08x", feature.preset, created);
|
||||
if (created != NGX_SUCCESS)
|
||||
{
|
||||
if (feature.cudaHandle != nullptr) g.ngx.releaseFeature(feature.cudaHandle);
|
||||
@@ -9062,9 +9083,11 @@ static NgxResult create_cuda_feature(Feature& feature)
|
||||
}
|
||||
}
|
||||
return created;
|
||||
});
|
||||
}, "CUDA feature creation");
|
||||
logf("NVSDK_NGX_CUDA_CreateFeature(feature=%u) -> 0x%08x",
|
||||
rayReconstruction ? NGX_FEATURE_RAY_RECONSTRUCTION : NGX_FEATURE_SUPER_SAMPLING, result);
|
||||
logf("CUDA_CREATE_END preset=%u result=0x%08x elapsed_ms=%.1f", feature.preset, result,
|
||||
std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - createStarted).count());
|
||||
if (result == NGX_SUCCESS) feature.cudaCreationDeferred = false;
|
||||
return result;
|
||||
}
|
||||
@@ -9339,6 +9362,33 @@ static void init_engine_vulkan()
|
||||
for (uint32_t index = 0; index < count; ++index)
|
||||
g_vk.float8 |= std::strcmp(extensions[index].extensionName, "VK_EXT_shader_float8") == 0;
|
||||
}
|
||||
auto features2 = reinterpret_cast<PFN_vkGetPhysicalDeviceFeatures2>(instanceProc(instance, "vkGetPhysicalDeviceFeatures2"));
|
||||
auto matrices = reinterpret_cast<PFN_vkGetPhysicalDeviceCooperativeMatrixPropertiesKHR>(
|
||||
instanceProc(instance, "vkGetPhysicalDeviceCooperativeMatrixPropertiesKHR"));
|
||||
bool fp8 = false;
|
||||
if (features2 && matrices)
|
||||
{
|
||||
VkPhysicalDeviceShaderFloat8FeaturesEXT f8 = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_FLOAT8_FEATURES_EXT};
|
||||
VkPhysicalDeviceCooperativeMatrixFeaturesKHR cm = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_COOPERATIVE_MATRIX_FEATURES_KHR};
|
||||
cm.pNext = g_vk.float8 ? &f8 : nullptr;
|
||||
VkPhysicalDeviceFeatures2 features = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2, &cm};
|
||||
features2(g_vk.physical, &features);
|
||||
uint32_t n = 0;
|
||||
if (cm.cooperativeMatrix && matrices(g_vk.physical, &n, nullptr) == VK_SUCCESS)
|
||||
{
|
||||
std::vector<VkCooperativeMatrixPropertiesKHR> properties(n, {VK_STRUCTURE_TYPE_COOPERATIVE_MATRIX_PROPERTIES_KHR});
|
||||
if (matrices(g_vk.physical, &n, properties.data()) == VK_SUCCESS)
|
||||
for (uint32_t i = 0; i < n; ++i)
|
||||
{
|
||||
g_vk.matrixF16 |= d4r_matrix_tuple(properties[i], VK_COMPONENT_TYPE_FLOAT16_KHR);
|
||||
fp8 |= d4r_matrix_tuple(properties[i], VK_COMPONENT_TYPE_FLOAT8_E4M3_EXT);
|
||||
}
|
||||
}
|
||||
fp8 = fp8 && f8.shaderFloat8 && f8.shaderFloat8CooperativeMatrix;
|
||||
}
|
||||
g_vk.float8 = g_vk.float8 && fp8;
|
||||
logf("engine backend: f16 matrix tuple %s; FP8 matrix tuple %s", g_vk.matrixF16 ? "supported" : "unsupported",
|
||||
g_vk.float8 ? "supported" : "unsupported");
|
||||
logf("engine backend: VK_EXT_shader_float8 %s", g_vk.float8 ? "available (M and L engine layers)" : "unavailable: "
|
||||
"M and L run on the CUDA backend");
|
||||
// the features vkd3d-proton created its device with
|
||||
@@ -10328,6 +10378,8 @@ static std::string rr_model_directory(int slot)
|
||||
char folder[MAX_PATH * 3] = {};
|
||||
WideCharToMultiByte(CP_ACP, 0, (g_portable.dir + folders[slot]).c_str(), -1, folder, sizeof(folder), nullptr, nullptr);
|
||||
directory = folder;
|
||||
if (slot < 2 && !g_vk.float8)
|
||||
directory += "-gfx11";
|
||||
}
|
||||
return directory[0] == '/' ? "Z:" + directory : directory;
|
||||
}
|
||||
@@ -10346,6 +10398,10 @@ static void init_engine_backend_rr(int slot)
|
||||
const auto start = std::chrono::steady_clock::now();
|
||||
try
|
||||
{
|
||||
std::string metadata;
|
||||
if (!read_whole_file(widen(path + "\\rr.txt"), metadata) ||
|
||||
!d4r_rr_model_compatible(metadata, g_vk.matrixF16, g_vk.float8))
|
||||
throw std::runtime_error("model arithmetic is unsupported by this GPU (no pipelines created)");
|
||||
auto model = std::make_unique<d4r::RModel>(g_vk.physical, g_vk.device, path, VK_NULL_HANDLE, g_vk.int64Atomics);
|
||||
if (model->preset() != g_rrPreset[slot])
|
||||
throw std::runtime_error("the directory holds a model of another preset");
|
||||
|
||||
@@ -620,6 +620,13 @@ static NgxResult native_vk_init(unsigned long long id, const wchar_t* path, VkIn
|
||||
VkPhysicalDeviceIDProperties identity = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_ID_PROPERTIES};
|
||||
VkPhysicalDeviceProperties2 properties = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2, &identity};
|
||||
properties2(physical, &properties);
|
||||
VkPhysicalDevicePCIBusInfoPropertiesEXT bus = {VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PCI_BUS_INFO_PROPERTIES_EXT};
|
||||
properties.pNext = &bus;
|
||||
properties2(physical, &properties);
|
||||
char pci[32] = {};
|
||||
if (bus.pciDomain || bus.pciBus || bus.pciDevice || bus.pciFunction)
|
||||
std::snprintf(pci, sizeof(pci), "%04x:%02x:%02x.%x", bus.pciDomain, bus.pciBus, bus.pciDevice, bus.pciFunction);
|
||||
SetEnvironmentVariableA("D4R_RENDER_PCI", pci[0] ? pci : nullptr);
|
||||
if (identity.deviceLUIDValid)
|
||||
{
|
||||
uint32_t low, high;
|
||||
|
||||
+137
-36
@@ -13,6 +13,8 @@
|
||||
#include <unistd.h>
|
||||
#include <time.h>
|
||||
#include "d4r_native_selection.h"
|
||||
#include "d4r_gpu_identity.h"
|
||||
#include "d4r_build.h"
|
||||
#include "d4r_motion_dilation.h"
|
||||
#include "d4r_native_motion.h"
|
||||
#include "d4r_wire_history.h"
|
||||
@@ -430,46 +432,54 @@ static void native_cleanup(void)
|
||||
remove_directory(native_served);
|
||||
}
|
||||
|
||||
/* The gfx target ("gfx1101") of the GPU DLSS runs on, "" when unknown: D4R_GPU_ARCH if set, else the KFD
|
||||
topology's GPU with the most SIMDs. With an integrated and a discrete GPU (Ryzen 7000/9000 desktops) the
|
||||
first KFD GPU node can be the integrated one, whose kernels would not match the discrete GPU. */
|
||||
/* Use the rendering adapter's PCI address, never the first/largest KFD GPU. Without an authenticated
|
||||
address a single GPU is unambiguous; multiple GPUs use translated kernels instead of a guessed target. */
|
||||
static void gpu_architecture(char* out, size_t size)
|
||||
{
|
||||
out[0] = '\0';
|
||||
const char* forced = getenv("D4R_GPU_ARCH");
|
||||
if (forced != NULL && strncmp(forced, "gfx", 3) == 0)
|
||||
char pci[64] = {}, only[32] = {};
|
||||
GetEnvironmentVariableA("D4R_RENDER_PCI", pci, sizeof(pci));
|
||||
int gpus = 0, matches = 0;
|
||||
DIR* nodes = opendir("/sys/class/kfd/kfd/topology/nodes");
|
||||
if (nodes == NULL) return;
|
||||
for (struct dirent* node; (node = readdir(nodes)) != NULL;)
|
||||
{
|
||||
snprintf(out, size, "%s", forced);
|
||||
return;
|
||||
}
|
||||
unsigned long best_simds = 0;
|
||||
int gpus = 0;
|
||||
for (int node = 0; node < 16; ++node)
|
||||
{
|
||||
char path[128];
|
||||
snprintf(path, sizeof(path), "/sys/class/kfd/kfd/topology/nodes/%d/properties", node);
|
||||
if (node->d_name[0] == '.') continue;
|
||||
char path[512];
|
||||
snprintf(path, sizeof(path), "/sys/class/kfd/kfd/topology/nodes/%s/properties", node->d_name);
|
||||
FILE* properties = fopen(path, "r");
|
||||
if (properties == NULL)
|
||||
continue;
|
||||
char line[256];
|
||||
unsigned long simds = 0, target = 0;
|
||||
unsigned long simds = 0, target = 0, domain = ~0ul, location = ~0ul;
|
||||
while (fgets(line, sizeof(line), properties) != NULL)
|
||||
{
|
||||
sscanf(line, "simd_count %lu", &simds);
|
||||
sscanf(line, "gfx_target_version %lu", &target);
|
||||
sscanf(line, "domain %lu", &domain);
|
||||
sscanf(line, "location_id %lu", &location);
|
||||
}
|
||||
fclose(properties);
|
||||
if (simds == 0 || target == 0)
|
||||
continue;
|
||||
++gpus;
|
||||
if (simds > best_simds)
|
||||
snprintf(only, sizeof(only), "gfx%lu%lu%lx", target / 10000, (target / 100) % 100, target % 100);
|
||||
if (d4r_pci_matches_kfd(pci, domain, location))
|
||||
{
|
||||
best_simds = simds;
|
||||
snprintf(out, size, "gfx%lu%lu%lx", target / 10000, (target / 100) % 100, target % 100);
|
||||
++matches;
|
||||
snprintf(out, size, "%s", only);
|
||||
}
|
||||
}
|
||||
if (gpus > 1)
|
||||
tracef("native kernels: %d GPUs; using the one with the most SIMDs (%s); D4R_GPU_ARCH overrides", gpus, out);
|
||||
closedir(nodes);
|
||||
if (!pci[0] && gpus == 1) snprintf(out, size, "%s", only);
|
||||
else if (matches != 1) out[0] = '\0';
|
||||
const char* forced = getenv("D4R_GPU_ARCH");
|
||||
if (forced != NULL && strcmp(forced, out) != 0)
|
||||
{
|
||||
tracef("D4R_GPU_ARCH=%s does not match the rendering GPU (%s); native kernels disabled", forced, out);
|
||||
out[0] = '\0';
|
||||
}
|
||||
tracef("GPU target: render PCI=%s KFD GPUs=%d matches=%d architecture=%s", pci, gpus, matches, out);
|
||||
}
|
||||
|
||||
static void prepare_native_kernels(const char* cache_home)
|
||||
@@ -480,6 +490,11 @@ static void prepare_native_kernels(const char* cache_home)
|
||||
/* a release directory holds one manifest and set of code objects per GPU target */
|
||||
char architecture[32], source[1024], path[1200];
|
||||
gpu_architecture(architecture, sizeof(architecture));
|
||||
if (!architecture[0])
|
||||
{
|
||||
unsetenv("D4R_ZLUDA_NATIVE_DIR");
|
||||
return;
|
||||
}
|
||||
FILE* manifest = NULL;
|
||||
/* RDNA4 with native FP8 WMMA (D4R_ZLUDA_WMMA_FP8_NATIVE=1, d4r.ini NativeFp8): the <target>-fp8 variant,
|
||||
whose kernels match ZLUDA's FP8 lowering; no other GPU has such a directory */
|
||||
@@ -1385,8 +1400,12 @@ static void tracef(const char* format, ...)
|
||||
vsnprintf(line, sizeof(line), format, args);
|
||||
va_end(args);
|
||||
|
||||
struct timespec now;
|
||||
clock_gettime(CLOCK_MONOTONIC, &now);
|
||||
const double seconds = (double)now.tv_sec + (double)now.tv_nsec / 1000000000.0;
|
||||
const unsigned long thread = GetCurrentThreadId();
|
||||
pthread_mutex_lock(&trace_lock);
|
||||
fprintf(stderr, "[d4r nvcuda] %s\n", line);
|
||||
fprintf(stderr, "[d4r nvcuda] [%.6f tid=%lu] %s\n", seconds, thread, line);
|
||||
fflush(stderr);
|
||||
|
||||
char path[MAX_PATH] = {0};
|
||||
@@ -1396,9 +1415,14 @@ static void tracef(const char* format, ...)
|
||||
FILE* output = fopen(path, "a");
|
||||
if (output != NULL)
|
||||
{
|
||||
fprintf(output, "%s\n", line);
|
||||
fprintf(output, "[%.6f tid=%lu] %s\n", seconds, thread, line);
|
||||
fclose(output);
|
||||
}
|
||||
else
|
||||
{
|
||||
static int warned;
|
||||
if (!warned++) fprintf(stderr, "[d4r nvcuda] cannot open D4R_CUDA_TRACE=%s: %s\n", path, strerror(errno));
|
||||
}
|
||||
}
|
||||
pthread_mutex_unlock(&trace_lock);
|
||||
}
|
||||
@@ -1473,7 +1497,8 @@ typedef CUresult(__attribute__((sysv_abi)) *CUGETERRORSTRING_FN)(CUresult, const
|
||||
typedef CUresult(__attribute__((sysv_abi)) *CUGETPROCADDRESS_FN)(const char*, void**, int, uint64_t, int*);
|
||||
|
||||
static pthread_once_t context_once = PTHREAD_ONCE_INIT;
|
||||
static CUresult context_setup_result = CUDA_ERROR_NOT_INITIALIZED;
|
||||
static CUresult context_bootstrap_result = CUDA_ERROR_NOT_INITIALIZED;
|
||||
static _Thread_local CUresult context_setup_result = CUDA_ERROR_NOT_INITIALIZED;
|
||||
static CUcontext cuda_context;
|
||||
|
||||
static unsigned int get_process_u32(const char* name, unsigned int fallback)
|
||||
@@ -1487,6 +1512,36 @@ static unsigned int get_process_u32(const char* name, unsigned int fallback)
|
||||
return end != value && *end == '\0' ? (unsigned int)parsed : fallback;
|
||||
}
|
||||
|
||||
/* This calls ZLUDA directly: bridge wrappers would recurse through context bootstrap. */
|
||||
static CUresult selected_cuda_device(CUdevice* selected)
|
||||
{
|
||||
CUDEVICEGETCOUNT_FN count_fn = (CUDEVICEGETCOUNT_FN)find_zluda_symbol("cuDeviceGetCount");
|
||||
CUDEVICEGET_FN get_fn = (CUDEVICEGET_FN)find_zluda_symbol("cuDeviceGet");
|
||||
typedef CUresult(__attribute__((sysv_abi)) * PCI_FN)(char*, int, CUdevice);
|
||||
PCI_FN pci_fn = (PCI_FN)find_zluda_symbol("cuDeviceGetPCIBusId");
|
||||
int count = 0;
|
||||
if (!count_fn || !get_fn || count_fn(&count) != CUDA_SUCCESS || count < 1)
|
||||
return CUDA_ERROR_NOT_INITIALIZED;
|
||||
char expected[64] = {};
|
||||
GetEnvironmentVariableA("D4R_RENDER_PCI", expected, sizeof(expected));
|
||||
if (!expected[0] && count == 1) return get_fn(selected, 0);
|
||||
if (!expected[0] || !pci_fn) return CUDA_ERROR_NOT_SUPPORTED;
|
||||
int matches = 0;
|
||||
for (int ordinal = 0; ordinal < count; ++ordinal)
|
||||
{
|
||||
CUdevice device;
|
||||
char actual[64] = {};
|
||||
if (get_fn(&device, ordinal) == CUDA_SUCCESS && pci_fn(actual, sizeof(actual), device) == CUDA_SUCCESS)
|
||||
{
|
||||
tracef("compute adapter ordinal=%d device=%d PCI=%s render=%s", ordinal, device, actual, expected);
|
||||
if (d4r_same_pci(expected, actual)) { *selected = device; ++matches; }
|
||||
}
|
||||
}
|
||||
if (matches != 1)
|
||||
tracef("CUDA adapter mapping refused: render PCI=%s matches=%d; no GPU will be guessed", expected, matches);
|
||||
return matches == 1 ? CUDA_SUCCESS : CUDA_ERROR_NOT_SUPPORTED;
|
||||
}
|
||||
|
||||
static void create_default_zluda_context(void)
|
||||
{
|
||||
CUINIT_FN init = (CUINIT_FN)find_zluda_symbol("cuInit");
|
||||
@@ -1499,30 +1554,30 @@ static void create_default_zluda_context(void)
|
||||
return;
|
||||
}
|
||||
|
||||
context_setup_result = init(0);
|
||||
if (context_setup_result != CUDA_SUCCESS)
|
||||
context_bootstrap_result = init(0);
|
||||
if (context_bootstrap_result != CUDA_SUCCESS)
|
||||
{
|
||||
tracef("cuInit returned %d", context_setup_result);
|
||||
tracef("cuInit returned %d", context_bootstrap_result);
|
||||
return;
|
||||
}
|
||||
|
||||
int device_count = 0;
|
||||
context_setup_result = get_device_count(&device_count);
|
||||
if (context_setup_result != CUDA_SUCCESS || device_count < 1)
|
||||
context_bootstrap_result = get_device_count(&device_count);
|
||||
if (context_bootstrap_result != CUDA_SUCCESS || device_count < 1)
|
||||
{
|
||||
tracef("cuDeviceGetCount returned %d, count %d", context_setup_result, device_count);
|
||||
if (context_setup_result == CUDA_SUCCESS)
|
||||
context_setup_result = CUDA_ERROR_NOT_INITIALIZED;
|
||||
tracef("cuDeviceGetCount returned %d, count %d", context_bootstrap_result, device_count);
|
||||
if (context_bootstrap_result == CUDA_SUCCESS)
|
||||
context_bootstrap_result = CUDA_ERROR_NOT_INITIALIZED;
|
||||
return;
|
||||
}
|
||||
|
||||
CUdevice device = 0;
|
||||
context_setup_result = get_device(&device, 0);
|
||||
if (context_setup_result == CUDA_SUCCESS)
|
||||
context_setup_result = create_context(&cuda_context, 0, device);
|
||||
if (context_setup_result != CUDA_SUCCESS)
|
||||
context_bootstrap_result = selected_cuda_device(&device);
|
||||
if (context_bootstrap_result == CUDA_SUCCESS)
|
||||
context_bootstrap_result = create_context(&cuda_context, 0, device);
|
||||
if (context_bootstrap_result != CUDA_SUCCESS)
|
||||
{
|
||||
tracef("context creation returned %d", context_setup_result);
|
||||
tracef("context creation returned %d", context_bootstrap_result);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1531,7 +1586,21 @@ static void create_default_zluda_context(void)
|
||||
|
||||
static void ensure_context(void)
|
||||
{
|
||||
typedef CUresult(__attribute__((sysv_abi)) * CURRENT_FN)(CUcontext*);
|
||||
CURRENT_FN current = (CURRENT_FN)find_zluda_symbol("cuCtxGetCurrent");
|
||||
CUcontext active = NULL;
|
||||
if (current && current(&active) == CUDA_SUCCESS && active != NULL)
|
||||
{
|
||||
context_setup_result = CUDA_SUCCESS;
|
||||
return; /* NGX already selected a context; do not replace it with a second context. */
|
||||
}
|
||||
pthread_once(&context_once, create_default_zluda_context);
|
||||
context_setup_result = context_bootstrap_result;
|
||||
if (context_setup_result == CUDA_SUCCESS && cuda_context != NULL)
|
||||
{
|
||||
CUCTX_CURRENT_FN set = (CUCTX_CURRENT_FN)find_zluda_symbol("cuCtxSetCurrent");
|
||||
if (set) context_setup_result = set(cuda_context);
|
||||
}
|
||||
}
|
||||
|
||||
/* D4R_NGX_STREAM_PRIORITY=high|normal (default off): the null-stream work of NGX, the shim and this bridge
|
||||
@@ -2567,6 +2636,7 @@ static int before_teardown(void)
|
||||
|
||||
CUresult WINAPI cuInit(unsigned int flags)
|
||||
{
|
||||
tracef("bridge build %s; CUDA_BEGIN api=cuInit flags=%u", D4R_RUNTIME_BUILD_ID, flags);
|
||||
CUINIT_FN function = (CUINIT_FN)find_zluda_symbol("cuInit");
|
||||
CUresult result = function != NULL ? function(flags) : CUDA_ERROR_NOT_INITIALIZED;
|
||||
tracef("cuInit flags=%u result=%d", flags, result);
|
||||
@@ -2618,6 +2688,14 @@ CUresult WINAPI cuCtxGetDevice(CUdevice* device)
|
||||
|
||||
CUresult WINAPI cuCtxCreate_v2(CUcontext* context, unsigned int flags, CUdevice device)
|
||||
{
|
||||
CUdevice selected = -1;
|
||||
CUresult mapping = selected_cuda_device(&selected);
|
||||
if (mapping != CUDA_SUCCESS || selected != device)
|
||||
{
|
||||
if (context) *context = NULL;
|
||||
tracef("CUDA context refused: device=%d mapped render device=%d status=%d", device, selected, mapping);
|
||||
return mapping == CUDA_SUCCESS ? CUDA_ERROR_NOT_SUPPORTED : mapping;
|
||||
}
|
||||
CUCTXCREATE_FN function = (CUCTXCREATE_FN)find_zluda_symbol("cuCtxCreate_v2");
|
||||
CUresult result = function != NULL ? function(context, flags, device) : CUDA_ERROR_NOT_INITIALIZED;
|
||||
tracef("cuCtxCreate_v2 flags=0x%x device=%d result=%d context=%p", flags, device,
|
||||
@@ -2659,6 +2737,7 @@ static int elide_ngx_sync(void)
|
||||
|
||||
CUresult WINAPI d4rCtxSynchronize(void)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=d4rCtxSynchronize");
|
||||
if (before_teardown())
|
||||
return CUDA_SUCCESS;
|
||||
CUCTX_SYNCHRONIZE_FN function = (CUCTX_SYNCHRONIZE_FN)find_zluda_symbol("cuCtxSynchronize");
|
||||
@@ -2671,6 +2750,7 @@ CUresult WINAPI d4rCtxSynchronize(void)
|
||||
|
||||
CUresult WINAPI cuCtxSynchronize(void)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=cuCtxSynchronize");
|
||||
if (before_teardown())
|
||||
return CUDA_SUCCESS;
|
||||
if (elide_ngx_sync())
|
||||
@@ -2695,6 +2775,15 @@ CUresult WINAPI cuDeviceGetLuid(char* luid, unsigned int* device_node_mask, CUde
|
||||
if ((result == CUDA_SUCCESS || result == CUDA_ERROR_NOT_SUPPORTED) &&
|
||||
luid != NULL && device_node_mask != NULL)
|
||||
{
|
||||
CUdevice selected = -1;
|
||||
CUresult mapping = selected_cuda_device(&selected);
|
||||
if (mapping != CUDA_SUCCESS || selected != device)
|
||||
{
|
||||
if (luid) memset(luid, 0, 8);
|
||||
if (device_node_mask) *device_node_mask = 0;
|
||||
tracef("cuDeviceGetLuid device=%d is not the mapped render adapter (selected=%d status=%d)", device, selected, mapping);
|
||||
return mapping == CUDA_SUCCESS ? CUDA_SUCCESS : mapping;
|
||||
}
|
||||
unsigned int low = get_process_u32("D4R_CUDA_LUID_LOW", 0xffffffffu);
|
||||
unsigned int high = get_process_u32("D4R_CUDA_LUID_HIGH", 0xffffffffu);
|
||||
unsigned int node_mask = get_process_u32("D4R_CUDA_NODE_MASK", 0xffffffffu);
|
||||
@@ -2791,6 +2880,7 @@ static const void* replacement_image(const void* image, const char* api)
|
||||
|
||||
CUresult WINAPI cuModuleLoadData(CUmodule* module, const void* image)
|
||||
{
|
||||
tracef("CUDA_BEGIN api=cuModuleLoadData image=%p", image);
|
||||
CUMODULELOADDATA_FN function = (CUMODULELOADDATA_FN)find_zluda_symbol("cuModuleLoadData");
|
||||
verify_native_kernels(image);
|
||||
capture_ptx_if_present(image, "cuModuleLoadData");
|
||||
@@ -2804,6 +2894,7 @@ CUresult WINAPI cuModuleLoadData(CUmodule* module, const void* image)
|
||||
CUresult WINAPI cuModuleLoadDataEx(CUmodule* module, const void* image, unsigned int option_count,
|
||||
int* options, void** option_values)
|
||||
{
|
||||
tracef("CUDA_BEGIN api=cuModuleLoadDataEx image=%p options=%u", image, option_count);
|
||||
CUMODULELOADDATAEX_FN function = (CUMODULELOADDATAEX_FN)find_zluda_symbol("cuModuleLoadDataEx");
|
||||
verify_native_kernels(image);
|
||||
capture_ptx_if_present(image, "cuModuleLoadDataEx");
|
||||
@@ -2817,6 +2908,7 @@ CUresult WINAPI cuModuleLoadDataEx(CUmodule* module, const void* image, unsigned
|
||||
|
||||
CUresult WINAPI cuModuleLoad(CUmodule* module, const char* filename)
|
||||
{
|
||||
tracef("CUDA_BEGIN api=cuModuleLoad file=%s", filename != NULL ? filename : "<null>");
|
||||
CUMODULELOAD_FN function = (CUMODULELOAD_FN)find_zluda_symbol("cuModuleLoad");
|
||||
CUresult result = function != NULL ? function(module, filename) : missing("cuModuleLoad");
|
||||
tracef("cuModuleLoad file=%s result=%d module=%p", filename != NULL ? filename : "<null>",
|
||||
@@ -2837,6 +2929,7 @@ CUresult WINAPI cuModuleUnload(CUmodule module)
|
||||
|
||||
CUresult WINAPI cuModuleGetFunction(CUfunction* function_out, CUmodule module, const char* name)
|
||||
{
|
||||
tracef("CUDA_BEGIN api=cuModuleGetFunction module=%p name=%s", module, name ? name : "<null>");
|
||||
CUMODULEGETFUNCTION_FN function = (CUMODULEGETFUNCTION_FN)find_zluda_symbol("cuModuleGetFunction");
|
||||
CUresult result = function != NULL ? function(function_out, module, name) : missing("cuModuleGetFunction");
|
||||
if (result == CUDA_SUCCESS && function_out != NULL)
|
||||
@@ -2882,6 +2975,7 @@ CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x,
|
||||
unsigned int block_z, unsigned int shared_bytes, CUstream stream,
|
||||
void** kernel_params, void** extra)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=cuLaunchKernel function=%p", function_handle);
|
||||
static CULAUNCHKERNEL_FN function;
|
||||
if (function == NULL)
|
||||
function = (CULAUNCHKERNEL_FN)find_zluda_symbol("cuLaunchKernel");
|
||||
@@ -2964,6 +3058,7 @@ CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x,
|
||||
|
||||
CUresult WINAPI cuMemAlloc(CUdeviceptr* pointer, size_t bytes)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=cuMemAlloc bytes=%zu", bytes);
|
||||
CUMEMALLOC_FN function = (CUMEMALLOC_FN)find_zluda_symbol("cuMemAlloc_v2");
|
||||
CUresult result = function != NULL ? function(pointer, bytes) : missing("cuMemAlloc_v2");
|
||||
if (result == CUDA_SUCCESS && pointer != NULL)
|
||||
@@ -3022,6 +3117,8 @@ CUresult WINAPI cuMemcpy2D(const void* copy)
|
||||
|
||||
CUresult WINAPI cuMemcpyHtoDAsync(CUdeviceptr destination, const void* source, size_t bytes, CUstream stream)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=cuMemcpyHtoDAsync bytes=%zu dst=0x%llx stream=%p", bytes,
|
||||
(unsigned long long)destination, stream);
|
||||
CUMEMCPYHTODASYNC_FN function = (CUMEMCPYHTODASYNC_FN)find_zluda_symbol("cuMemcpyHtoDAsync_v2");
|
||||
if (microcuda_k_write) microcuda_k_write(destination, bytes);
|
||||
stream = d4r_ngx_stream(stream);
|
||||
@@ -3033,6 +3130,7 @@ CUresult WINAPI cuMemcpyHtoDAsync(CUdeviceptr destination, const void* source, s
|
||||
|
||||
CUresult WINAPI cuMemcpyDtoH(void* destination, CUdeviceptr source, size_t bytes)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=cuMemcpyDtoH bytes=%zu src=0x%llx", bytes, (unsigned long long)source);
|
||||
CUMEMCPYDTOH_FN function = (CUMEMCPYDTOH_FN)find_zluda_symbol("cuMemcpyDtoH_v2");
|
||||
CUresult result = function != NULL ? function(destination, source, bytes) : missing("cuMemcpyDtoH_v2");
|
||||
TRACE_CALL(result, "cuMemcpyDtoH->v2 dst=%p src=0x%llx bytes=%zu result=%d",
|
||||
@@ -3378,6 +3476,7 @@ static int d4r_forward_is_teardown(const char* target)
|
||||
#define D4R_FORWARD(name, target, count, params, args) \
|
||||
CUresult WINAPI name params \
|
||||
{ \
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=" #name); \
|
||||
if (d4r_forward_is_teardown(#target) && before_teardown()) \
|
||||
return CUDA_SUCCESS; \
|
||||
typedef CUresult(__attribute__((sysv_abi)) * function_type) params; \
|
||||
@@ -3394,6 +3493,7 @@ D4R_FORWARD(cuEventRecord, cuEventRecord, 2, (D4rArg a0, D4rArg a1), (a0, (D4rAr
|
||||
D4R_FORWARD(cuEventRecordWithFlags, cuEventRecordWithFlags, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, (D4rArg)d4r_ngx_stream((CUstream)a1), a2))
|
||||
CUresult WINAPI cuEventSynchronize(D4rArg a0)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=cuEventSynchronize event=%p", (void*)a0);
|
||||
if (before_teardown())
|
||||
return CUDA_SUCCESS;
|
||||
typedef CUresult(__attribute__((sysv_abi)) * function_type)(D4rArg);
|
||||
@@ -3411,6 +3511,7 @@ CUresult WINAPI cuEventSynchronize(D4rArg a0)
|
||||
game's split command list only after this event has actually completed. */
|
||||
CUresult WINAPI d4rEventSynchronize(D4rArg a0)
|
||||
{
|
||||
if (trace_verbose()) tracef("CUDA_BEGIN api=d4rEventSynchronize event=%p", (void*)a0);
|
||||
if (before_teardown())
|
||||
return CUDA_SUCCESS;
|
||||
typedef CUresult(__attribute__((sysv_abi)) * function_type)(D4rArg);
|
||||
|
||||
Reference in new issue
Block a user