engine: cut the models' and the engines' memory

- the weights go into the command buffer with vkCmdUpdateBuffer (recordContents); the host-visible staging
  copies are gone, and the host copies are dropped once recorded
- an engine's own output image is created only for frames that do not write the caller's texture
  (K, M, RR D/E, RR F), cleared on first use
- preset F: the tile slots share the arena with the network's regions, and the gather records' rows fill the
  gaps between encoder 0's regions (rr.txt "s3", six words on enc0's block; D4R_ENGINE_RR_ARENA keeps the
  old separate layout for arena dumps)
- the shim reserves the render size the feature was created with for the M/L Vulkan network instead of the
  output size; a frame beyond it goes to a second, output-sized engine (1.1 GB less for M, 1.4 for L at
  1280x720 -> 4K); replay_m gains D4R_M_CAPACITY

Every reference capture still byte-identical; through the shim the harness process peaks 159 MiB lower.
This commit is contained in:
woutermans committed 2026-10-14 15:14:28 +02:00
1 parent 162bbd77c8
commit af450a92b9
10 files changed
+237 -93

No files matched your search

+28 -1
View File
@@ -312,7 +312,7 @@ how the passes were arranged:
- **Encoder 0 in two passes** (-0.2 ms). One shader ran NVIDIA's 32-lane gather on one of the block's four waves
while the other three waited: 0.40 ms for the gather, 0.24 ms as a pass of its own in 32-thread groups.
`gen_rr2_enc0.py --split gather | block` writes both; the gather pass (`stage enc0g`, the same bindings and
parameter block as enc0) stores each window's record in the arena after the tile slots: the first product's A
parameter block as enc0) stores each window's record in the arena buffer (see Memory below): the first product's A
fragments (1024 words) and, per lane, the values the tail reads from the gather (six binary16 on the reference
build; the generator finds them and recomputes the tail's index arithmetic). The record's address is a word
compile_rr2.py appends to enc0's argument block. Cost: 76 bytes per render pixel of arena.
@@ -354,6 +354,33 @@ scripts/run_rr_engine_compare.sh`, frames aligned by one):
moving scene that showed as 40 - 50 dB on later frames in a synthetic scene whose affected feature is rounding
noise around zero in both paths; no other configuration was affected.
### Memory (2026-10-14)
At 1280x720 -> 4K the engine needs about 600 MB for preset F; the two 1.5x output histories (2 x 149 MB, NGX has
the same) are half of it. NVIDIA's DLL, its allocations traced through the CUDA path and replayed on an RTX PRO 4500
Blackwell, takes 526 MiB of feature state, 106 MiB of input / output arrays and 2 MiB of code, plus 262 MiB of CUDA
context. Three cuts, every capture still byte-identical (`verify.sh`, `verify_x.sh`) and the shim's frames
byte-identical to the previous build, same speed:
- **The tile slots share the arena with the network's regions.** Every arena region is written and read within one
frame; in the output stage only the network's output (dec0_1's argument 64) and something after it are read, so
the slots that fit before the network's output go there (all 8 at 720p -> 4K) and the rest after the arena. A
first try that put the slots right after the network's output broke 360p -> 1080p's output stage.
- **The gather records' rows of windows fill the gaps between encoder 0's regions**: the two largest (units
1216 - 2240 and 2304 - 2560) and then from the end of its last region (unit 2816) on, instead of one piece after
the tile slots. The records (76 bytes a render pixel, about 2450 units) are larger than all the gaps, so 46 of 93
rows at 720p still reach past the arena (36 MB). Models from compile_rr2.py mark this with "s3" on the enc0g line
and append six words to enc0's block (three ranges' addresses, the first two's rows; `gen_rr2_enc0.py`); older
models keep the one-piece layout. Same speed as one piece (four alternating runs at 720p -> 4K).
- **The engine's own output image is created only for frames that do not write the game's texture**, and the
weights go into the command buffer with `vkCmdUpdateBuffer` instead of through a host-visible staging copy that
stayed allocated.
Through the shim (direct output) the harness process peaks 159 MiB lower (1100 -> 941 MiB). Not exact, so not
done: the pooled histories as RGB10A2 (`D4R_RR2_POOLED=rgb10_a2`, -70 MB, see above); they hold q / 1023, which
neither binary16 nor UNORM16 represents exactly. `D4R_ENGINE_RR_ARENA`
keeps the old separate layout, so that arena dumps show the network's regions as NGX leaves them.
## Against a real NVIDIA card: `D4R_RR_FUSE` (2026-10-13)
Every reference above is the CUDA path through ZLUDA. NVIDIA's own kernels (310.9.1's `rr2_*` PTX, replayed through
+10 -2
View File
@@ -117,8 +117,15 @@ def split_layout(xf_word, pre, tail, f16vars):
if not (v in f16vars or re.fullmatch(r'r\d+', v)):
raise SystemExit(f'the tail reads {v} from the gather: not a 16- or 32-bit register')
record = 1024 + 32 * len(saved)
# the records' rows of windows lie in up to three ranges of the arena (the runtime's choice, where the network's
# regions are dead during encoder 0): words XF_WORD, +2, +4 their byte addresses, +1 and +3 the first two's rows
where = (f' const uint rr2_gx = (param.w[0] + (param.w[20] & 0xffffu) + 7u) / 8u;\n'
f' const uint rr2_rec = param.w[{xf_word}u] / 4u + {record}u * (gl_WorkGroupID.y * (param.w[0] / 8u + 1u) + gl_WorkGroupID.x);\n')
f' uint rr2_y = gl_WorkGroupID.y, rr2_base = param.w[{xf_word}u];\n'
f' if (rr2_y >= param.w[{xf_word + 1}u]) {{\n'
f' rr2_y -= param.w[{xf_word + 1}u]; rr2_base = param.w[{xf_word + 2}u];\n'
f' if (rr2_y >= param.w[{xf_word + 3}u]) {{ rr2_y -= param.w[{xf_word + 3}u]; rr2_base = param.w[{xf_word + 4}u]; }}\n'
f' }}\n'
f' const uint rr2_rec = rr2_base / 4u + {record}u * (rr2_y * (param.w[0] / 8u + 1u) + gl_WorkGroupID.x);\n')
return recomputed, saved, record, where
@@ -128,7 +135,8 @@ def gather_pass(xf_word, head, decls, pre, tail, first, f16vars):
One shader holding both runs the gather on one of the block's four waves while the other three wait, which
costs more than the record's trip through memory (RX 9070 XT, 1280x720, 2026-10-13: gather 0.40 -> 0.24 ms).
The record: 1024 words of A fragments ([token slot][64 FP8 feature slots], as the block's XF), then per lane the
gather's values the tail reads. Its address: parameter word XF_WORD (bytes) + 4 * RECORD * (y * (W / 8 + 1) + x)."""
gather's values the tail reads. Its address: the byte address of row y's range (parameter words XF_WORD to
XF_WORD + 4) + 4 * RECORD * (y' * (W / 8 + 1) + x), y' the row within its range."""
recomputed, saved, record, where = split_layout(xf_word, pre, tail, f16vars)
var = lambda r: r.lstrip('%')
store = [' if (rr2_valid) { // the first product\'s A fragments: lane 4g + t, register q -> row, 4 k',