308 lines
18 KiB
Markdown
308 lines
18 KiB
Markdown
# Windows RDNA4 measurements
|
||
|
||
Hardware: RX 9070 XT / gfx1201, Windows 11, display driver 32.0.31041.1004.
|
||
Runtime: TheRock 10.2.0a20260929, HIP 7.17.26386; ZLUDA b0161a4 with
|
||
`D4R_ZLUDA_WMMA=1`, FP8 widening, native FP8 disabled, f16 reference rounding.
|
||
|
||
The optional profiler uses HIP events on the actual launch stream, including
|
||
separate measurements of one-time weight preparation. It synchronizes each
|
||
sampled launch and changes scheduling: these numbers are not application FPS.
|
||
HIP occupancy predictions are not measured hardware occupancy. The native
|
||
objects' actual wave32, LDS, VGPR/SGPR and spill counts are available in
|
||
`llvm-readobj --notes` and can be attached to the JSON report. Memory bandwidth
|
||
and WMMA utilization were not measured; no utilization is inferred from timing.
|
||
|
||
The identical four-frame standalone D3D12/OptiScaler workloads output 512x288.
|
||
Summing the measured main-kernel event times and dividing by four gives:
|
||
|
||
| Preset | Fully translated, ms/frame | Native FP16-equivalent, ms/frame |
|
||
| --- | ---: | ---: |
|
||
| K | 157.745441 | 0.684675 |
|
||
| M | 7.428538 | 1.566500 |
|
||
|
||
Both rows include translated input/output tails. Native overrides cover all
|
||
11 K layers and five M layers; M's tube layer executes six times per frame.
|
||
M native and translated RGB are bit-exact, with max absolute/relative error 0.
|
||
K native remains bit-exact against its independently validated native control;
|
||
fully translated K has the previously documented arithmetic differences.
|
||
All outputs are finite. These are reference-correct codegen settings, not
|
||
measurements of a relaxed fast-math translated configuration.
|
||
|
||
M's first profiled Evaluate includes about 0.9 seconds of host initialization;
|
||
the four-sample CPU mean must not be treated as steady-state latency. Its
|
||
median Evaluate host completion is 4.855 ms. GPU events and prep phases are
|
||
reported separately in the raw JSON.
|
||
|
||
The initial game runner used PowerShell line-oriented output redirection.
|
||
It heavily throttled verbose stdout/stderr. The runner now concurrently drains
|
||
raw byte streams using the same approach as the standalone diagnostics.
|
||
|
||
| Silent Hill 2 stage, CPU mean | Initial capture, ms | Raw-byte capture with HIP profiler, ms |
|
||
| --- | ---: | ---: |
|
||
| Input fence wait | 5.641 | 0.833 |
|
||
| Input conversion and CUDA array upload | 13.196 | 0.668 |
|
||
| NGX Evaluate host call | 25.715 | 3.350 |
|
||
| NGX all-stream completion | 1.846 | 0.033 |
|
||
| Output CUDA array download | 4.288 | 0.167 |
|
||
| Output conversion and shared fence signal | 1.500 | 0.033 |
|
||
|
||
The first run additionally scanned every output (10.142 ms in that throttled
|
||
capture); the kernel-profiling run disables that scan. Both use a temporary
|
||
1280x720 window and current-frame VRAM interop. These are separate menu/game
|
||
runs, not a controlled whole-game FPS comparison. No synchronization was
|
||
removed to obtain the improvement. The raw-byte profiling run completed
|
||
2345 K frames, with no d4r failure, CPU image copy or previous-frame output.
|
||
|
||
Reproduce native or translated measurements with locally supplied NVIDIA DLLs
|
||
at the repository root and the separately built profiling runtime:
|
||
|
||
```powershell
|
||
.\scripts\windows\profile-windows-rdna4.ps1 -Preset 11
|
||
.\scripts\windows\profile-windows-rdna4.ps1 -Preset 11 -Translated
|
||
.\scripts\windows\profile-windows-rdna4.ps1 -Preset 13
|
||
.\scripts\windows\profile-windows-rdna4.ps1 -Preset 13 -Translated
|
||
```
|
||
|
||
`profile_report.py <results-directory> --output profile.json` summarizes raw
|
||
CPU stages and every kernel's event time, launch geometry, registers, private
|
||
memory, LDS and predicted blocks per multiprocessor. Add
|
||
`--metadata-directory <directory-of-llvm-readobj-notes>` for native wave sizes
|
||
and compiler spill counts. Reports and proprietary workload inputs remain local.
|
||
|
||
Native FP8 remains a separate optional experiment; the package uses the
|
||
validated FP16-equivalent baseline. Its later checks and timings are below.
|
||
|
||
## K host synchronization and asynchronous submission
|
||
|
||
Current priority is K performance. The user's synchronous 4K reading is
|
||
49–51 FPS / 53% GPU utilization, versus approximately 80+ FPS / 100% with
|
||
FSR4. These readings do not identify a CPU hardware limit or provide a
|
||
controlled same-scene comparison. M/FP8 optimization is deferred.
|
||
|
||
ZLUDA candidate `a1c506f` adds `D4R_ZLUDA_PROFILE_API`, enabled by the game
|
||
runner's `-ProfileCudaApi`. It measures existing API host durations, aggregates
|
||
per thread and logs every two seconds without HIP events or additional waits.
|
||
The synchronous control shows five `cuCtxSynchronize` calls per frame with
|
||
about 3.9 ms aggregate host time. CUDA launch host overhead is approximately
|
||
0.46 ms per frame. Cached copy-list setup is only 0.011 ms median. These host
|
||
measurements include GPU waits and must not be added to GPU kernel times.
|
||
|
||
| K/4K command submission, CPU median | Synchronous | First async | Async without CUDA mutex |
|
||
| --- | ---: | ---: | ---: |
|
||
| Entire submission, ms | 10.512 | 2.676 | 0.181 |
|
||
| DLSS boundary, ms | 10.083 | 2.604 | 0.124 |
|
||
| Consumer suffix completion, ms | 0.350 | 0.005 | 0.004 |
|
||
|
||
The first async game run completes 9481 finite frames; the user reports
|
||
60 FPS / 63% GPU. The mutex-free run crashes during feature recreation after
|
||
366 finite frames, so its submission timing is not a successful gameplay
|
||
benchmark. The standalone burst/recreate reproducer subsequently identifies
|
||
caller-owned GPU backing freed after ReleaseFeature returns. Waiting for
|
||
pending D3D12 consumers in ReleaseFeature fixes the reproducer: all twelve
|
||
sync/async test frames pass, and corresponding RGB outputs are exact. Game
|
||
stability and FPS must be checked separately on the fixed build.
|
||
|
||
The fixed game retest on 2026-10-02 completes 16941 K/4K frames, all finite,
|
||
including feature recreation and color-format reimports, with zero backend
|
||
failures. The user reports 62 FPS / 63% GPU. Median CPU submission is
|
||
0.197 ms, NGX host completion on the worker is 4.400 ms and output validation
|
||
is 0.380 ms. The 10.162 ms median worker input wait includes queued game
|
||
rendering. These results establish a stable five-minute test of this fix,
|
||
not a conclusion about the remaining GPU idle time or FSR4 performance.
|
||
|
||
`-AsyncInterop` is opt-in. It queues input/output fences on the submission
|
||
thread, executes CUDA on a worker and retires owned command memory by GPU
|
||
completion. Three copy slots bound outstanding ownership. The queue consumes
|
||
the output of the same frame, with no CPU image copy. The worker's input wait
|
||
includes its lead behind submitted rendering and is not a CPU submission
|
||
stall. Neither that duration nor the NGX recording interval measures Present
|
||
latency. Reconfiguration waits for all consumers; ordinary submission does not
|
||
hold the CUDA mutex. The experimental runtime currently supports one queue.
|
||
|
||
```powershell
|
||
.\scripts\windows\test-async-k.ps1
|
||
```
|
||
|
||
The test generates fresh synchronous controls, submits three frames without
|
||
intermediate CPU completion, and repeats while releasing/recreating the DLSS
|
||
feature with work pending. Results include exact RGB comparisons, finite GPU
|
||
scans, stdout/stderr, exceptions and runtime provenance. To include the local
|
||
validated output-store optimization, pass its combined native directory with
|
||
`-NativeRoot`; proprietary objects remain local.
|
||
|
||
`-ProfileGpuBoundary` (requires `-AsyncInterop`) adds D3D12 timestamps before
|
||
and after the VRAM input/output copies. Resolved results are read only once
|
||
the copy slot's existing fence completes; 32 diagnostic bytes per frame are
|
||
read and no new completion wait is added. `external_span_ms` spans HIP and
|
||
host/driver scheduling across the external fence. `between_boundaries_ms`
|
||
also includes game rendering, and `input_interval_ms` is not Present timing.
|
||
These queries obey the queue's timestamp ordering and add GPU commands; they
|
||
are disabled for an ordinary FPS run. GPU clock behavior during idle can
|
||
affect intervals; no hardware occupancy is inferred. Timestamp semantics:
|
||
[Microsoft D3D12 timing](https://learn.microsoft.com/en-us/windows/win32/direct3d12/timing).
|
||
|
||
The timestamp-enabled 4K burst/recreation gate completes twelve frames, all
|
||
finite; all six async RGB outputs still match their fresh controls exactly.
|
||
Each async mode emits all three timing records, including the final frame
|
||
at feature release. Results: `test-results/k-async-gpu-boundary-4k`.
|
||
All sixteen CTest gates pass on the rebuilt shim. Reproduce with:
|
||
|
||
```powershell
|
||
.\scripts\windows\test-async-k.ps1 -ProfileGpuBoundary
|
||
```
|
||
|
||
Further game coverage (HIP kernel profiling disabled, GPU output checks enabled):
|
||
M at 1920x1080 completes 4833 frames, all finite. K at 3840x2160 completes
|
||
4685 frames, all finite. Neither has a backend failure, CPU image copy or an
|
||
aged-frame fallback. Their mean interop/NGX CPU stages total about 13.6 and
|
||
10.1 ms respectively; these are different scenes, not a K/M comparison.
|
||
|
||
The no-debugger K/4K run completes 5774 frames, with 5774 finite output scans
|
||
and 63514 native transformer launches. The user reports 38-42 FPS. Additional
|
||
CPU measurements exclude replay/publication as the main cause of low FPS:
|
||
|
||
| CPU measurement | Mean, ms | Median, ms |
|
||
| --- | ---: | ---: |
|
||
| Command split setup | 0.284 | 0.275 |
|
||
| State replay within split setup | 0.025 | 0.023 |
|
||
| Submitted texture state publication | 0.029 | 0.029 |
|
||
| Consumer suffix completion | 0.405 | 0.409 |
|
||
| NGX recording interval | 27.474 | 28.466 |
|
||
|
||
Recording intervals are between NGX recording calls on the same thread, not
|
||
DXGI Present times. The interval includes rendering and application scheduling.
|
||
A second K/4K run with the attached debugger completes 3567 frames without
|
||
failure, with median recording interval 30.091 ms. The runs include different
|
||
menu/game states; this does not establish a precise debugger FPS speedup.
|
||
Its sparse debugger events also rule out a per-kernel OutputDebugString storm.
|
||
An earlier control that stayed on the startup screen completed zero DLSS frames
|
||
and is excluded. Normal game launches now omit the attached debugger; crash
|
||
diagnostics remain available with `-CaptureExceptions`.
|
||
|
||
The 2961-frame 4K K profile identifies the translated output tail as its largest
|
||
kernel: 4.401670 ms mean GPU time, versus about 2.875 ms for all eleven native
|
||
transformer layers combined. Ported upstream's texture-store recipe to native
|
||
Windows. It retains accuracy mode, wave32, FP16 subnormals and f16 reference
|
||
accumulation; native FP8 remains disabled.
|
||
|
||
Both output variants execute on every checked frame and match the previous
|
||
RGB output bit-exactly: eight consecutive 512x288 frames per variant, then
|
||
three consecutive 3840x2160 frames per variant. All RGBA components are finite.
|
||
Identical synthetic inputs and public NGX flags give these 4K event times
|
||
(three samples each; host initialization and readback are outside the event):
|
||
|
||
| K output variant | Translated mean, ms | Native stores mean, ms |
|
||
| --- | ---: | ---: |
|
||
| LDR, regular depth, high-resolution MV | 2.907823 | 1.803233 |
|
||
| HDR, inverted depth, low-resolution MV | 2.978547 | 2.088347 |
|
||
|
||
A subsequent game run executes 6221 frames with the native HDR output tail,
|
||
zero backend failures and mean tail GPU time 1.801918 ms. The earlier game
|
||
profile's 4.401670 ms is from a different scene/scheduling state; use the
|
||
identical-input table for the controlled comparison, not an inferred FPS
|
||
multiplier. This profiled game run did not enable the NaN scan.
|
||
|
||
Reproduce the private texture build and full-frame comparisons:
|
||
|
||
```powershell
|
||
.\scripts\windows\build-native-texture.ps1 -DlssDll "$PWD\nvngx_dlss.dll"
|
||
.\scripts\windows\build-native-texture.ps1 -DlssDll "$PWD\nvngx_dlss.dll" -Kernel hiluma_engine_output_depthreg_mvhi_ldr_max_v2_rel
|
||
.\scripts\windows\test-native-texture.ps1 -OutputResolution 3840x2160 -Iterations 3
|
||
```
|
||
|
||
These objects contain code derived from the locally supplied NVIDIA PTX. They
|
||
stay in ignored local directories and are excluded from the public package.
|
||
The installer accepts them with `-LocalTextureKernels` only after validation,
|
||
checking the exact DLL identity, object and manifest hashes. M retains its
|
||
independently validated FP16-equivalent baseline.
|
||
|
||
The following no-debugger K/4K run completes 9982 frames and 9982 finite GPU
|
||
scans. Kernel event profiling is disabled; the validated private output tail
|
||
executes natively. The user reports 43-47 FPS. Mean NGX recording interval is
|
||
21.867 ms. No backend failure, CPU image copy or previous-frame output occurs.
|
||
|
||
Random one-in-64 hook samples then identify contention in the shared routing
|
||
table used by every intercepted D3D12 command. Per-thread weak lookup caches
|
||
now bypass that table on repeated calls to a list. Table mutations invalidate
|
||
cache generations; Reset/address reuse cannot route to a previous suffix, and
|
||
thread caches do not retain game resources. The per-list and submission locks
|
||
remain. A GPU regression resets and re-splits the exact same list with changed
|
||
root constants twice per iteration, checking both prefix and suffix results.
|
||
|
||
| Sampled command-hook access | Shared table | Per-thread weak cache |
|
||
| --- | ---: | ---: |
|
||
| Busiest recording thread, mean us/call | 0.203 | 0.066 |
|
||
| Estimated access + capture across threads, ms/s | 977.528 | 233.606 |
|
||
|
||
The latter estimate includes lock waits summed across concurrently running
|
||
threads, not CPU execution time or serial frame latency. The two game runs
|
||
have different thread populations (31 versus 29); there is no identical frame
|
||
trace. Generated forwarding methods have separate driver/capture markers;
|
||
special-case hooks contribute access samples but lack those body markers.
|
||
|
||
With the cache, 4770 complete K/4K frames and finite GPU scans pass, with zero
|
||
backend failures, CPU image copies or previous-frame output. Median NGX
|
||
recording interval is 19.473 ms. The user reports 49-51 FPS and 53% GPU usage
|
||
in the previously measured scene. These user readings are separate from the
|
||
sampled hook estimates and do not establish isolated WMMA utilization.
|
||
All sixteen CTest gates pass, including 64 same-object Reset/split checks in
|
||
each physical command-backend test and the software indirect-root regression.
|
||
Use `-ProfileCommandHooks -ProfileStages` to collect samples; the sampler is
|
||
disabled for ordinary play.
|
||
|
||
Native M FP8 was subsequently compiled and tested on this gfx1201. ISA confirms
|
||
`v_wmma_f32_16x16x16_fp8_fp8`, with packed e4m3 weight images; exact FP16
|
||
attention/PV remains. All logical outputs and merges in forty recorded launches
|
||
match the previously NumPy-validated FP16 baseline byte-for-byte. Full DLSS RGB
|
||
also matches exactly: four 512x288 LDR frames and four 3840x2160 HDR frames,
|
||
all RGBA finite. This establishes correctness for those fixtures, not a speedup.
|
||
|
||
The original software operand packing makes the FP8 network slower in the
|
||
four-frame 4K test: sum of five main-kernel event means (tube weighted six times)
|
||
is 27.789640 ms versus FP16's 23.550875 ms. A paired in-process replay confirms
|
||
the regression without relying solely on separate process timings. Both modules
|
||
and prepared weights remain resident, 512 launches warm the GPU, then sixty-four
|
||
AB/BA pairs time the same restored input using HIP events. Diagnostic CPU input
|
||
restoration is outside the timed event and is unrelated to game VRAM interop.
|
||
|
||
Added opt-in `D4R_FP8_HW_PACK`: native packed conversion of already quantised
|
||
operands replaces software exponent/subnormal reconstruction. It preserves all
|
||
254 finite OCP e4m3 encodings, including signed zero, in an exhaustive GPU check.
|
||
All forty recorded launches still match exactly. Four complete 4K HDR frames
|
||
also match FP16 RGB exactly with finite RGBA. ISA confirms both packed converts
|
||
and FP8 WMMA. The baseline build's sixteen objects remain byte-identical.
|
||
|
||
| Paired replay median, ms | FP16 (software-packing test) | FP8 software packing | FP16 (hardware-packing test) | FP8 hardware packing |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| enc1 | 0.175450 | 0.217050 | 0.175100 | 0.177700 |
|
||
| enc2 | 0.141350 | 0.164550 | 0.132550 | 0.131550 |
|
||
| enc3 tube | 0.108000 | 0.112000 | 0.107600 | 0.099850 |
|
||
| dec2 | 0.129900 | 0.146000 | 0.143250 | 0.140850 |
|
||
| dec1 | 0.169650 | 0.194450 | 0.154250 | 0.156550 |
|
||
|
||
These are two paired tests on the captured 512x288 workload. Medians avoid large
|
||
isolated scheduling outliers; raw per-pair ratios are retained. They identify a
|
||
substantial cost in software packing but do not establish an overall FPS gain.
|
||
The separately run four-frame 4K hardware-packing network mean is 24.603463 ms;
|
||
it still does not beat the measured FP16 total. Default packaging retains FP16.
|
||
Hardware counters for bandwidth or WMMA utilization were not measured.
|
||
|
||
Reproduce the isolated build, encoding/replay checks and paired timings using
|
||
the existing private validated captures; no source editing is required:
|
||
|
||
```powershell
|
||
.\scripts\windows\build-native-m-fp8.ps1 -DlssDll "$PWD\nvngx_dlss.dll" -HardwarePacking
|
||
.\scripts\windows\test-native-m-fp8.ps1 -ModuleDirectory "$PWD\build\native-m-fp8-hardware-pack" -CaptureDirectory "$PWD\test-results\ngx-m-exact-pv-capture\captures" -BaselineValidation "$PWD\test-results\m-native-exact-pv-temporal-reference\summary.json" -Benchmark
|
||
```
|
||
|
||
Omit `-HardwarePacking` to compile the original FP8 experiment. The replay
|
||
validator stops at the first mismatch, retains GPU stdout/stderr/exit code and
|
||
hashes every compared logical output. `profile_report.py` accepts replay logs
|
||
and reports per-pair GPU time and candidate/control ratios. This experiment is
|
||
not installed by the game runner or substituted silently for the baseline.
|
||
The conversion builtin is listed in the official
|
||
[Clang AMDGPU builtin reference](https://clang.llvm.org/docs/AMDGPUBuiltinReference.html#builtin-amdgcn-cvt-pk-fp8-f32).
|
||
|
||
References: [HIP events](https://rocmdocs.amd.com/projects/HIP/en/develop/doxygen/html/group___event.html),
|
||
[HIP occupancy API](https://rocm.docs.amd.com/projects/HIP/en/latest/doxygen/html/group___occupancy.html).
|