shim: adaptive marker polling, opt-in GPU-side input wait/completion and RGBA8 output

- [Interop] MarkerPollUs = auto is the new default: the input-marker wait learns when
  each feature's inputs become ready, sleeps in <=200 us chunks, yields near the
  prediction and backs off when late (d4r_input_wait.h); a number keeps fixed pacing
  (0 yields). d4r_config.py no longer exports 200 when the key is unset

- D4R_SHIM_GPU_INPUT_WAIT=1 (experimental, split VRAM, age 0): NGX's null stream
  waits on a coherent Vulkan word imported into HIP (d4rStreamWaitValue32Eq) instead
  of CPU polling. The original readback marker stays the authority; a CPU watchdog
  releases discarded lists and timeouts. Denoiser, verify, capture and dilation
  frames keep the CPU wait

- D4R_SHIM_GPU_COMPLETION=1 (needs GPU_INPUT_WAIT): the bridge writes a completion
  word right after the native output kernel (d4rNgxWriteValue32), and the game's queue
  polls it with a bounded compute wait (kernels/support/vulkan_wait.comp), replacing
  the split list's CPU timeline signal. frame_retired writes the same word on every
  failure path

- D4R_SHIM_NATIVE_COMPLETION=1: a null-stream callback in the bridge signals the
  split timeline (d4rNgxTimeline*); D4rFrameCompletion::can_signal only releases the
  oldest outstanding frame

- RGBA8 output: tex_common.h adds an 'R4' redirect tag, so K's native output kernel
  stores packed RGBA8 directly (D4R_SHIM_OUTPUT_RGBA8=1, checked against the kernel
  manifest by d4rOutputKernelRgba8). Host packing handles RGBA8 too.
  patches/vkd3d-proton/0004 (VKD3D_D4R_LINEAR_RGBA8=WxH) makes such outputs linear and
  exportable so the result can be written in place

- D4R_SHIM_FUSE_RGBA8_INPUT=1: one Vulkan pass converts and copies RGBA8 color input
  (vulkan_read.comp mode 5 matches the blit's toward-zero halves, plus a row stride)

- profiling: thread CPU time from the Unix bridge (Wine's GetThreadTimes ticks at
  10 ms); game-queue D3D12 timestamps (vk_input/vk_dlss_wait/vk_output/vk_total);
  gpu_eval split into input_gpu_wait and dlss_gpu; D4R_PROFILE_GPU_EVENTS,
  _TIMESTAMPS, _LOG and _CPU_TIME=0 switch off individual parts for bisection

- bridge: D4R_NGX_STREAM_PRIORITY=high|normal runs NGX on a priority stream
  (off by default; no gain measured)

- release_feature: a feature already marked retiring by a failed drain can still
  be released normally

- harness: RGBA8 with motion and jitter scenes (re-quantized each frame),
  D4R_HARNESS_RGBA8_RAMP, D4R_HARNESS_THRASH_MB to evict L2/Infinity Cache before
  each evaluation

- kernels/native: in_hoist.py and split_output_paths.py (opt-in scheduling
  experiments); scripts/build_k_ldr_output.sh builds K's LDR output with an
  optional D4R_LDR_OUTPUT_MAX_VGPR cap

- tests: test_input_wait.py and test_latency_optimizations.py; MarkerPollUs and
  frame-completion tests updated

config/d4r.ini.default

@@ -97,18 +97,19 @@ VramInterop = true
 InputSync = true
 ; Faster hand-offs between the game and DLSS (same frame, same result). Need FrameAge = 0 and VRAM
 ; interop, and a runtime built from this checkout (scripts/install_d4r_runtime.sh).
-; Input readiness is checked with CPU sleeps; recorded-but-discarded lists never queue GPU waits.
+; Input readiness uses adaptive CPU polling by default. D4R_SHIM_GPU_INPUT_WAIT=1 enables experimental
+; null-stream GPU waits for same-frame Super Sampling; diagnostics and host input reads retain CPU waits.
 ;   LinearInputs: DLSS reads the game's inputs in place once the input marker is ready
 ;   DirectOutput: the result is written straight into the game-side buffer (preset K, and only while
 ;     NGX runs the native output kernel; drops NVIDIA's "DLSS SDK" watermark)
 ;   ElideNgxSync: removes NGX's internal CPU waits between its GPU stages
 ;   EvalSync = false: no CPU wait between DLSS and the output copy
-;   MarkerPollUs: how often (microseconds) the CPU checks for the game's inputs
+;   MarkerPollUs: auto learns readiness and yields near it; N fixes the poll interval in microseconds
 LinearInputs = true
 DirectOutput = true
 ElideNgxSync = true
 EvalSync = false
-MarkerPollUs = 200
+MarkerPollUs = auto
 
 [Engine]
 ; Experimental. Runs DLSS 4 and 4.5 (models K, L and M) on d4r's own Vulkan inference engine instead of NVIDIA's

kernels/native/in_hoist.py

@@ -0,0 +1,118 @@
+#!/usr/bin/env python3
+"""Issue independent texture fetches of the translated hiluma input kernel earlier (opt-in experiment).
+
+usage: in_hoist.py KERNEL.hip ANCHOR_REG HANDLE_REG [HANDLE_REG ...] > OUT.hip
+
+The input kernel (ptx2hip --ifconv --sink + in_lds2reg.py) is one straight-line body. Its 3x3 color neighbourhood
+fetches depend only on the thread's pixel position and kernel parameters, yet the compiler issues them after the
+depth -> motion -> history chain, adding serial memory round trips to a latency-bound kernel. This pass moves every
+fetch block `{ ::f4 t_ = tex_fetch(rdX, ...); ... }` whose handle register is one of HANDLE_REG's copies, together
+with the backward slice of statements computing its operands, to right after the statement that assigns
+ANCHOR_REG, followed by a scheduling barrier (instruction order only). Nothing else moves and no statement
+changes, so every value is computed exactly as before.
+
+Refuses unless: every register the moved statements assign is assigned exactly once in the kernel; every register
+they read is assigned before the anchor (or by another moved statement, earlier in order); there is no memory store
+and no control flow (label/goto/return) between the anchor and the last moved statement.
+"""
+import re
+import sys
+
+REG = re.compile(r'\b(?:rd|rs|r|f|p|shv\d+_|dpv\d+_)\d+\b|\bl\d+_\w+\b')
+ASSIGN = re.compile(r'^\s*([A-Za-z_]\w*)\s*=(?!=)')
+
+
+def stmts(body_lines):
+    """group lines into statements: fetch blocks '{ ... }' become one statement"""
+    out, i = [], 0
+    while i < len(body_lines):
+        line = body_lines[i]
+        if line.strip().startswith('{ ::f4 t_ ='):
+            j = i
+            while body_lines[j].strip() != '}':
+                j += 1
+            out.append(body_lines[i:j + 1])
+            i = j + 1
+        else:
+            out.append([line])
+            i += 1
+    return out
+
+
+def defs_uses(st):
+    text = ''.join(st)
+    if text.strip().startswith('{ ::f4 t_ ='):
+        defs = set(re.findall(r'^\s*([A-Za-z_]\w*) = t_\[\d\];', text, re.M))
+        call = text.split(';', 1)[0].replace('::f4', '')
+        return defs, set(REG.findall(call))
+    m = ASSIGN.match(text)
+    if m:
+        lhs = m.group(1)
+        rhs = text[m.end():]
+        return {lhs}, set(REG.findall(rhs))
+    return set(), set(REG.findall(text))
+
+
+def main():
+    src = open(sys.argv[1]).read().split('\n')
+    anchor, handles = sys.argv[2], set(sys.argv[3:])
+    start = next(i for i, l in enumerate(src) if l.startswith('extern "C" __attribute__((global))')) + 2
+    end = max(i for i, l in enumerate(src) if l.strip() == '}')
+    S = stmts([l + '\n' for l in src[start:end]])
+    info = [defs_uses(s) for s in S]
+    assigned = {}
+    for k, (d, _) in enumerate(info):
+        for r in d:
+            assigned.setdefault(r, []).append(k)
+    # handle copies: rdY = (uint64_t)(rdX) chains from the given handle registers
+    alias = set(handles)
+    for k, s in enumerate(S):
+        m = re.match(r'^\s*(rd\d+) = \(uint64_t\)\((rd\d+)\);$', s[0])
+        if m and m.group(2) in alias:
+            alias.add(m.group(1))
+    ia = [k for k, (d, _) in enumerate(info) if anchor in d]
+    if len(ia) != 1:
+        raise SystemExit(f'anchor {anchor} assigned {len(ia)} times')
+    ia = ia[0]
+    fetches = [k for k, s in enumerate(S) if s[0].strip().startswith('{ ::f4 t_ = tex_fetch(')
+               and re.search(r'tex_fetch\((rd\d+)', s[0]).group(1) in alias and k > ia]
+    if not fetches:
+        raise SystemExit('no fetch to move')
+    need, move = set(), set(fetches)
+    for k in fetches:
+        need |= info[k][1]
+    while need:
+        r = need.pop()
+        ks = assigned.get(r, [])
+        if not ks:
+            continue  # parameters / builtins
+        if len(ks) != 1:
+            raise SystemExit(f'register {r} is assigned {len(ks)} times')
+        k = ks[0]
+        if k <= ia or k in move:
+            continue
+        move.add(k)
+        need |= info[k][1]
+    last = max(move)
+    for k in range(ia + 1, last + 1):
+        text = ''.join(S[k])
+        if re.search(r'^\s*\*\(|\bgoto\b|\breturn\b|^L__BB', text, re.M):
+            raise SystemExit(f'store or control flow between anchor and moved code: {text.strip()[:80]}')
+    for k in sorted(move):
+        for r in info[k][0]:
+            if len(assigned[r]) != 1:
+                raise SystemExit(f'moved register {r} assigned more than once')
+        for r in info[k][1]:
+            ks = assigned.get(r, [])
+            if ks and not (ks[0] <= ia or ks[0] in move and ks[0] < k):
+                raise SystemExit(f'moved statement reads {r} defined later')
+    # a scheduling barrier keeps the compiler from sinking the fetches back to their uses (order only)
+    barrier = ['    __builtin_amdgcn_sched_barrier(0); // in_hoist: fetches above are issued here\n']
+    body = ''.join(''.join(S[k]) for k in range(0, ia + 1)) + ''.join(''.join(S[k]) for k in sorted(move)) + \
+        ''.join(barrier) + ''.join(''.join(S[k]) for k in range(ia + 1, len(S)) if k not in move)
+    sys.stderr.write(f'in_hoist: moved {len(move)} statements ({len(fetches)} fetches) after {anchor}\n')
+    sys.stdout.write('\n'.join(src[:start]) + '\n' + body + '\n'.join(src[end:]))
+
+
+if __name__ == '__main__':
+    main()

kernels/native/split_output_paths.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""Split a translated hiluma output kernel into its two per-block paths (opt-in experiment).
+
+usage: split_output_paths.py KERNEL.hip > KERNEL_split.hip
+
+NVIDIA's hiluma_engine_output_* kernels pick one of two code paths per block: path B (an LDS tile of the
+block's render footprint) when the footprint is under 24 texels in both axes, which holds for every block at
+Quality .. Ultra Performance, else path A (direct fetches, DLAA). The choice depends only on the block index and
+kernel parameters, so it is uniform per block, and blocks share nothing. Path A needs more registers than path B
+(LDR mvlo: 101 vs 88 VGPRs), and one kernel carries the larger allocation for every block.
+
+This pass emits the same translated body twice:
+  NAME        runs path B and returns where path A would start;
+  NAME_post1  runs path A and returns where path B would start.
+The ZLUDA/MicroCUDA native hook launches NAME_post1 right after NAME on the same stream with the same parameters
+and grid (d4r_post1_grid_x/_y = 0), with the 16x16 blocks the PTX hard-codes (tid.y << 4). Each block executes
+exactly the instructions it executed before; only the block order across the two paths changes, which no output
+depends on (blocks never read what other blocks write in the same launch).
+
+The pass refuses anything but the expected entry structure:
+    pA = ((int32_t)rX < (int32_t)24);  pB = ((int32_t)rY < (int32_t)24);  pC = (pA && pB);
+    if (pC) goto LB;
+    goto LA;
+"""
+import re
+import sys
+
+
+def split(source):
+    head, sep, main = source.partition('extern "C" __attribute__((global))')
+    if not sep or 'extern "C" __attribute__((global))' in main:
+        raise SystemExit('expected exactly one kernel')
+    m = re.match(r'\s*__attribute__\(\(amdgpu_flat_work_group_size\(1, 256\)\)\) void (\w+)\((\w+)\)', main)
+    if not m:
+        raise SystemExit('unexpected kernel signature')
+    name = m.group(1)
+    pattern = re.compile(r'^    (p\d+) = \(\(int32_t\)r\d+ < \(int32_t\)24\);\n'
+                         r'    (p\d+) = \(\(int32_t\)r\d+ < \(int32_t\)24\);\n'
+                         r'    (p\d+) = \((p\d+) && (p\d+)\);\n'
+                         r'    if \((p\d+)\) goto (L__BB0_\d+);\n'
+                         r'    goto (L__BB0_\d+);\n', re.M)
+    found = list(pattern.finditer(main))
+    if len(found) != 1:
+        raise SystemExit(f'path selection found {len(found)} times, expected once')
+    g = found[0]
+    pa, pb, pc, x, y, pc2, path_b, path_a = g.groups()
+    if {x, y} != {pa, pb} or pc != pc2:
+        raise SystemExit('path selection predicate has an unexpected form')
+    # the predicate must not be reassigned anywhere (it is the block's path choice)
+    if len(re.findall(rf'^    {pc} = ', main, re.M)) != 1:
+        raise SystemExit(f'{pc} is assigned more than once')
+    sel = g.group(0)
+    lines = sel.splitlines(keepends=True)
+    main_b = main.replace(sel, ''.join(lines[:4]) + '    return; // path A blocks: NAME_post1\n', 1)
+    main_a = main.replace(sel, ''.join(lines[:3]) + f'    if ({pc}) return; // path B blocks: NAME\n'
+                          + f'    goto {path_a};\n', 1)
+    main_a = main_a.replace(f' void {name}(', f' void {name}_post1(', 1)
+    globals_ = ('\n// d4r native hook: launch NAME_post1 after NAME with the launch grid and the PTX\'s 16x16 blocks\n'
+                'extern "C" __attribute__((device, used)) uint32_t d4r_post1_grid_x = 0;\n'
+                'extern "C" __attribute__((device, used)) uint32_t d4r_post1_grid_y = 0;\n'
+                'extern "C" __attribute__((device, used)) uint32_t d4r_post1_block_x = 16;\n'
+                'extern "C" __attribute__((device, used)) uint32_t d4r_post1_block_y = 16;\n'
+                'extern "C" __attribute__((device, used)) uint32_t d4r_post1_block_z = 1;\n')
+    banner = '// split_output_paths.py: NAME = path B only, NAME_post1 = path A only (per-block choice unchanged)\n'
+    return (banner + head + globals_ + sep + main_b + '\n' + sep + main_a, path_b, path_a)
+
+
+if __name__ == '__main__':
+    text, b, a = split(open(sys.argv[1]).read())
+    sys.stderr.write(f'split: path B {b}, path A {a}\n')
+    sys.stdout.write(text)

kernels/support/vulkan_read.comp

@@ -2,7 +2,7 @@
 layout(local_size_x=8, local_size_y=8) in;
 layout(set=0, binding=0) uniform sampler2D sourceImage;
 layout(std430, set=0, binding=1) writeonly buffer Pixels { uint pixels[]; };
-layout(push_constant) uniform Size { uint width; uint height; uint mode; } size;
+layout(push_constant) uniform Size { uint width; uint height; uint mode; uint stride; } size;
 // Match the CPU canonical conversion, including round-to-even and subnormals.
 uint halfBits(float value) {
     uint bits = floatBitsToUint(value);
@@ -26,14 +26,24 @@ uint halfBits(float value) {
     if (remainder > halfway || (remainder == halfway && (m & 1u) != 0u)) ++result;
     return result;
 }
+// Mode 5 is only for RGBA8 UNORM: values are zero or normal positive halves. Preserve
+// the blit's toward-zero conversion; other modes keep the canonical round-to-even path.
+uint halfRtz(float value) {
+    uint b = floatBitsToUint(value);
+    if (value == 0.0) return 0u;
+    return (((b >> 23) - 112u) << 10) | ((b & 0x7fffffu) >> 13);
+}
 void main() {
     uvec2 p = gl_GlobalInvocationID.xy;
     if (p.x >= size.width || p.y >= size.height) return;
     ivec2 extent = textureSize(sourceImage, 0);
-    uint index = p.y * size.width + p.x;
+    uint index = p.y * size.stride + p.x;
     vec4 value = all(lessThan(ivec2(p), extent))
         ? texelFetch(sourceImage, ivec2(p), 0) : vec4(0.0);
-    if (size.mode == 1u) { // canonical RGBA16F color
+    if (size.mode == 5u) {
+        pixels[index * 2u] = halfRtz(value.x) | (halfRtz(value.y) << 16);
+        pixels[index * 2u + 1u] = halfRtz(value.z) | (halfRtz(value.w) << 16);
+    } else if (size.mode == 1u) { // canonical RGBA16F color
         pixels[index * 2u] = halfBits(value.x) | (halfBits(value.y) << 16);
         pixels[index * 2u + 1u] = halfBits(value.z) | (halfBits(value.w) << 16);
     } else if (size.mode == 3u) { // canonical RG16F motion

kernels/support/vulkan_wait.comp

@@ -0,0 +1,12 @@
+#version 450
+// D4R_SHIM_GPU_COMPLETION: holds the game's queue until DLSS has written this frame's completion word (HIP
+// hipStreamWriteValue32 after the output, or the CPU on every failure path). Bounded: after `limit` polls the
+// shader returns regardless, so a lost write can never hang the GPU (the frame then shows its previous output).
+layout(local_size_x = 1) in;
+layout(std430, set = 0, binding = 0) coherent volatile readonly buffer Words { uint words[]; };
+layout(push_constant) uniform Wait { uint index; uint value; uint limit; } wait;
+void main() {
+    for (uint poll = 0u; poll < wait.limit; ++poll)
+        if (words[wait.index] == wait.value)
+            return;
+}

kernels/tex/tex_common.h

@@ -371,15 +371,19 @@ extern "C" __attribute__((device)) int __ockl_image_width_2D(tsharp_t* image);
 extern "C" __attribute__((device)) int __ockl_image_height_2D(tsharp_t* image);
 __attribute__((device)) static inline uint32_t redirect_pitch(uint64_t surface)
 {
-    // 'R2' (RGBA16F texels) or 'R3' (R10G10B10A2_UNORM texels) tag in the high half (other handles may have
+    // 'R2' (RGBA16F), 'R3' (RGB10A2) or 'R4' (RGBA8) tag in the high half (other handles may have
     // data there), row pitch / 8 in the low half
     const uint32_t word = ((const __attribute__((address_space(4))) uint32_t*)surface)[21];
-    return (word >> 16) == 0x5232u || (word >> 16) == 0x5233u ? (word & 0xffffu) << 3 : 0u;
+    return (word >> 16) >= 0x5232u && (word >> 16) <= 0x5234u ? (word & 0xffffu) << 3 : 0u;
 }
 __attribute__((device)) static inline bool redirect_rgb10a2(uint64_t surface)
 {
     return (((const __attribute__((address_space(4))) uint32_t*)surface)[21] >> 16) == 0x5233u;
 }
+__attribute__((device)) static inline bool redirect_rgba8(uint64_t surface)
+{
+    return (((const __attribute__((address_space(4))) uint32_t*)surface)[21] >> 16) == 0x5234u;
+}
 // One channel as the game-side path produces it: the f32 value rounded toward zero to a half (the RGBA16F
 // store), then converted to UNORM by a blit (NaN and negatives 0, clamped at 1, rounded to nearest).
 __attribute__((device)) static inline uint32_t redirect_unorm(uint16_t half, float scale)
@@ -422,6 +426,10 @@ DEV __attribute__((always_inline)) void d4r_sust_p_v4b32(uint64_t surface, int32
                 *D4R_REDIRECT_PTR(uint32_t, redirect_row(surface, y) + 4 * x) =
                     redirect_unorm((uint16_t)lo, 1023.0f) | redirect_unorm((uint16_t)(lo >> 16), 1023.0f) << 10 |
                     redirect_unorm((uint16_t)hi, 1023.0f) << 20 | redirect_unorm((uint16_t)(hi >> 16), 3.0f) << 30;
+            else if (redirect_rgba8(surface))
+                *D4R_REDIRECT_PTR(uint32_t, redirect_row(surface, y) + 4 * x) =
+                    redirect_unorm((uint16_t)lo, 255.0f) | redirect_unorm((uint16_t)(lo >> 16), 255.0f) << 8 |
+                    redirect_unorm((uint16_t)hi, 255.0f) << 16 | redirect_unorm((uint16_t)(hi >> 16), 255.0f) << 24;
             else
                 *D4R_REDIRECT_PTR(uint2_t, redirect_row(surface, y) + 8 * x) = (uint2_t){lo, hi};
         }
@@ -526,9 +534,15 @@ DEV __attribute__((always_inline)) void d4r_sust_b32(uint64_t surface, int32_t x
     tsharp_t* image = (tsharp_t*)surface;
     if (redirect_pitch(surface) != 0)
     {
+        // Raw RGBA16F stores carry two half channels at each four-byte offset. Packing those channels
+        // separately keeps the original store ownership (no read-modify-write or cross-thread race).
+        if (redirect_rgba8(surface) && x >= 0 && (x & 3) == 0 && redirect_inside(surface, x >> 3, y))
+            *D4R_REDIRECT_PTR(uint16_t, redirect_row(surface, y) + (x >> 1)) =
+                (uint16_t)(redirect_unorm((uint16_t)data, 255.0f) |
+                           redirect_unorm((uint16_t)(data >> 16), 255.0f) << 8);
         // raw bits at byte offset x (RGBA16F rows are 8 bytes per pixel); the shim only asks for packed
-        // RGB10A2 texels from kernels without raw stores
+        // RGB10A2 texels from kernels without raw stores; RGBA8 preserves each half-pair store
-        if (!redirect_rgb10a2(surface) && x >= 0 && (x & 3) == 0 && redirect_inside(surface, x >> 3, y))
+        if (!redirect_rgb10a2(surface) && !redirect_rgba8(surface) && x >= 0 && (x & 3) == 0 && redirect_inside(surface, x >> 3, y))
             *D4R_REDIRECT_PTR(uint32_t, redirect_row(surface, y) + x) = data;
         return;
     }

packaging/d4r.ini

@@ -62,10 +62,10 @@ NativeFp8 = on
 ; Requires render- or display-resolution, unjittered motion vectors and same-frame linear VRAM inputs. Restart to change.
 ; Unsupported features keep original vectors; larger radii may affect disocclusion edges.
 MotionVectorDilation = 0
-; Keep inputs and output in video memory. Input and output readiness waits use CPU sleeps.
+; Keep inputs and output in video memory. Readiness uses adaptive CPU polling by default.
 VramInterop = true
-; Microseconds between checks for the game's input marker (default 200; 0 yields).
+; auto: adaptive input-marker polling; N: fixed interval in microseconds (0 yields).
-MarkerPollUs = 200
+MarkerPollUs = auto
 LinearInputs = true
 ElideNgxSync = true
 ; Lets DLSS's last kernel write straight into the game's output buffer (DLSS 4 / K only). It takes

patches/vkd3d-proton/0004-linear-rgba8-output.patch

@@ -0,0 +1,35 @@
+vkd3d-proton (d4r, opt-in): VKD3D_D4R_LINEAR_RGBA8=WxH also creates RGBA8 UNORM textures of exactly that size with
+unordered access linear and exportable (with VKD3D_D4R_LINEAR_TEXTURES=1), so the d4r shim can write DLSS's packed
+RGBA8 output in place (D4R_SHIM_INPLACE=1) instead of copying it. Every selected texture is logged (warn).
+Applies on top of 0001-0003.
+
+diff --git a/libs/vkd3d/resource.c b/libs/vkd3d/resource.c
+--- a/libs/vkd3d/resource.c
++++ b/libs/vkd3d/resource.c
+@@ -720,7 +720,25 @@
+             desc->DepthOrArraySize != 1 || desc->SampleDesc.Count != 1 ||
+             (desc->Flags & D3D12_RESOURCE_FLAG_ALLOW_DEPTH_STENCIL))
+         return false;
+-    return format->vk_format == VK_FORMAT_R16G16B16A16_SFLOAT || format->vk_format == VK_FORMAT_A2B10G10R10_UNORM_PACK32;
++    if (format->vk_format == VK_FORMAT_R16G16B16A16_SFLOAT || format->vk_format == VK_FORMAT_A2B10G10R10_UNORM_PACK32)
++        return true;
++    /* VKD3D_D4R_LINEAR_RGBA8=WxH (opt-in, with VKD3D_D4R_LINEAR_TEXTURES=1): also RGBA8 UNORM textures of exactly
++     * that size with unordered access, the shape of a DLSS output, so the shim can write DLSS's packed RGBA8 result
++     * in place. Other RGBA8 targets keep optimal tiling. Every selected texture is logged. */
++    if (format->vk_format == VK_FORMAT_R8G8B8A8_UNORM && (desc->Flags & D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS))
++    {
++        static unsigned int selected;
++        unsigned int width = 0, height = 0;
++        char size[32];
++        if (vkd3d_get_env_var("VKD3D_D4R_LINEAR_RGBA8", size, sizeof(size)) &&
++                sscanf(size, "%ux%u", &width, &height) == 2 && desc->Width == width && desc->Height == height)
++        {
++            WARN("d4r: linear exportable RGBA8 texture #%u (%ux%u, flags %#x).\n", ++selected, width, height,
++                    desc->Flags);
++            return true;
++        }
++    }
++    return false;
+ }
+ 
+ static HRESULT vkd3d_get_image_create_info(struct d3d12_device *device,

scripts/build_k_ldr_output.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Compile K's LDR output using the same PTX semantics as the HDR native route. The packed-store
+# sidecar authenticates the produced code bytes; it is not a substitute for temporal validation.
+set -euo pipefail
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+OUT="$(realpath -m "${1:?usage: build_k_ldr_output.sh OUT_DIR [KERNEL]}")"
+NAME="${2:-hiluma_engine_output_depthinv_mvlo_ldr_max_v2_rel}"
+[[ "$NAME" =~ ^hiluma_engine_output_depth(inv|reg)_mv(hi|lo)_ldr_max_v2_rel$ ]] || exit 2
+: "${D4R_DLSS_DLL:?set D4R_DLSS_DLL to the matching NVIDIA DLSS library}"
+mkdir -p "$OUT"
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+python3 "$ROOT/kernels/tools/extract_dlss_ptx.py" "$D4R_DLSS_DLL" "$WORK/ptx" >/dev/null
+PTX="$(rg -l "\.entry[[:space:]]+$NAME[[:space:]]*\(" "$WORK/ptx"/*.ptx)"
+[[ -n "$PTX" && "$PTX" != *$'\n'* ]] || { echo 'kernel must appear in exactly one module' >&2; exit 2; }
+# LDR uses another CFG/parameter layout than HDR: do not apply HDR's tile rewrite by register name.
+python3 "$ROOT/kernels/native/ptx2hip.py" --as1 "$PTX" "$NAME" > "$WORK/output.hip"
+# D4R_LDR_OUTPUT_MAX_VGPR=N (opt-in): cap the kernel's VGPRs. The DLAA path (path A, 100 VGPRs alone) otherwise sets
+# the allocation of every block (101 VGPRs, 7 waves/SIMD); with 96 its spills stay in path A and Quality..UP blocks
+# (path B, 88 VGPRs alone) run at 8 waves/SIMD: depthinv_mvlo output 0.825 -> 0.776 ms at 1440p, DLAA 0.840 ->
+# 0.812, every frame byte-identical (private/notes/k1440-implementation.md, O1). Lower caps spill in path B.
+if [[ -n "${D4R_LDR_OUTPUT_MAX_VGPR:-}" ]]; then
+    [[ "$D4R_LDR_OUTPUT_MAX_VGPR" =~ ^[0-9]+$ ]] || { echo 'D4R_LDR_OUTPUT_MAX_VGPR must be a number' >&2; exit 2; }
+    sed -i "s/__attribute__((amdgpu_flat_work_group_size(1, 256))) void $NAME(/__attribute__((amdgpu_flat_work_group_size(1, 256), amdgpu_num_vgpr($D4R_LDR_OUTPUT_MAX_VGPR))) void $NAME(/" "$WORK/output.hip"
+    grep -q "amdgpu_num_vgpr($D4R_LDR_OUTPUT_MAX_VGPR)" "$WORK/output.hip" || { echo 'VGPR cap not applied' >&2; exit 2; }
+fi
+bash "$ROOT/kernels/native/build_native.sh" "$WORK/output.hip" "$OUT/$NAME.hsaco" \
+    -DD4R_NATIVE_F2I -DD4R_AS1_STORES
+python3 - "$OUT" "$NAME" <<'PY'
+from pathlib import Path
+import sys
+out, name = Path(sys.argv[1]), sys.argv[2]
+value = 14695981039346656037
+for byte in (out / (name + '.hsaco')).read_bytes():
+    value = ((value ^ byte) * 1099511628211) & ((1 << 64) - 1)
+path = out / 'd4r-rgba8.txt'
+rows = path.read_text().splitlines() if path.exists() else ['# RGBA8 store capability: NAME FNV1a64 of code object']
+rows = [row for row in rows if not row.startswith(name + ' ')]
+path.write_text('\n'.join(rows + [f'{name} {value:016x}']) + '\n')
+PY
+python3 "$ROOT/kernels/tools/kernel_manifest.py" "$OUT" "$D4R_DLSS_DLL"

scripts/build_vulkan_staging.sh

@@ -4,6 +4,7 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 OUT="${1:-$ROOT/build/vulkan-staging}"
 mkdir -p "$OUT"
 glslangValidator -V --target-env vulkan1.0 -o "$OUT/read.spv" "$ROOT/kernels/support/vulkan_read.comp" >/dev/null
+glslangValidator -V --target-env vulkan1.0 -o "$OUT/wait.spv" "$ROOT/kernels/support/vulkan_wait.comp" >/dev/null
 for format in rgba16f rgba32f r11f_g11f_b10f rgb10_a2 rgba8; do
     glslangValidator -V --target-env vulkan1.0 -DOUTPUT_FORMAT="$format" \
         -o "$OUT/write_$format.spv" "$ROOT/kernels/support/vulkan_write.comp" >/dev/null

scripts/d4r_config.py

@@ -161,7 +161,7 @@ def main(argv):
         value = flag("Interop", key)
         if value is not None:
             env[var] = "1" if value else "0"
-    value = get("Interop", "MarkerPollUs") or "200"
+    value = get("Interop", "MarkerPollUs")
     dilation = get("Interop", "MotionVectorDilation")
     if dilation is not None and dilation not in ("0", "1", "2"):
         fail("[Interop] MotionVectorDilation must be 0, 1 or 2")

tests/test_frame_completion.py

@@ -30,8 +30,10 @@ struct Feature {
     int splitSemaphore = 1;
     bool engineNetTimeline = false; // the Vulkan timeline of this test, not the native network's
     int engineNet = 0;
+    void* ngxTimeline = nullptr;
 };
 static struct { int (*signalTimeline)(int, uint64_t) = nullptr; } g_engineNet;
+static struct { struct { int (*timelineSignal)(void*, uint64_t) = nullptr; } cu; } g;
 static uint64_t gpuTimeline = 0;
 static bool failSignal = false;
 static int signal(int, const VkSemaphoreSignalInfo* info) {

tests/test_input_wait.py

@@ -0,0 +1,188 @@
+"""CPU-only input-wait cancellation, polling and profiling contracts. No HIP or Wine runs."""
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def function(source, signature):
+    start = source.index(signature)
+    opening = source.index('{', start)
+    depth = 1
+    end = opening + 1
+    while depth:
+        depth += (source[end] == '{') - (source[end] == '}')
+        end += 1
+    return source[start:end]
+
+
+def run(source):
+    with tempfile.TemporaryDirectory() as tmp:
+        runner = Path(tmp) / 'input-wait'
+        subprocess.run(['c++', '-x', 'c++', '-std=c++20', '-pthread', '-Wall', '-Wextra', '-Werror',
+                        '-I', str(ROOT / 'tools'), '-o', str(runner), '-'],
+                       input=source, text=True, check=True, capture_output=True)
+        subprocess.run([str(runner)], check=True, timeout=5)
+
+
+class InputWaitTests(unittest.TestCase):
+    def test_bridge_queues_exact_null_stream_wait_or_reports_unsupported(self):
+        bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
+        wait = function(bridge, 'CUresult WINAPI d4rStreamWaitValue32Eq(')
+        run(r'''
+#include <cstdint>
+#include <cstring>
+#include <cassert>
+#define WINAPI
+using CUresult = int;
+using CUdeviceptr = uint64_t;
+using HIP_STREAM_WAIT_VALUE32_FN = int(*)(void*, void*, uint32_t, unsigned, uint32_t);
+constexpr int CUDA_SUCCESS = 0, CUDA_ERROR_NOT_SUPPORTED = 801;
+int context_setup_result = 0, supported = 1, waitResult = 0, queued = 0;
+bool hasWait = true;
+void ensure_context() {}
+void* d4r_ngx_stream(void* stream) { return stream; } // D4R_NGX_STREAM_PRIORITY off
+void tracef(const char*, ...) {}
+bool trace_verbose() { return false; }
+int device(int* out) { *out = 3; return 0; }
+int attribute(int* out, int attr, int dev) { assert(attr == 10013 && dev == 3); *out = supported; return 0; }
+int queue(void* stream, void* ptr, uint32_t value, unsigned flags, uint32_t mask) {
+    assert(stream == nullptr && ptr == reinterpret_cast<void*>(0x10008));
+    assert(value == 0xffffffffu && flags == 1 && mask == 0xffffffffu);
+    if (waitResult == 0) ++queued;
+    return waitResult;
+}
+void* hip_symbol(const char* name) {
+    if (strcmp(name, "hipStreamWaitValue32") == 0) return hasWait ? reinterpret_cast<void*>(queue) : nullptr;
+    if (strcmp(name, "hipDeviceGetAttribute") == 0) return reinterpret_cast<void*>(attribute);
+    if (strcmp(name, "hipGetDevice") == 0) return reinterpret_cast<void*>(device);
+    return nullptr;
+}
+''' + wait + r'''
+int main() {
+    assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 0 && queued == 1);
+    supported = 0;
+    assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 801 && queued == 1);
+    supported = 1; hasWait = false;
+    assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 801 && queued == 1);
+    hasWait = true; waitResult = 1;
+    assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 801 && queued == 1);
+    context_setup_result = 700;
+    assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 700 && queued == 1);
+}
+''')
+
+    def test_readiness_discard_retirement_timeout_wrap_and_adaptive_pacing(self):
+        run(r'''
+#include "d4r_input_wait.h"
+#include <cassert>
+int main() {
+    using A = D4rInputWaitAbort;
+    assert(d4r_input_wait_abort(7, 6, 6, false, 100) == A::None);
+    assert(d4r_input_wait_abort(7, 6, 8, false, 100) == A::Discarded);
+    assert(d4r_input_wait_abort(7, 7, 8, false, 100) == A::None); // slot recheck wins
+    assert(d4r_input_wait_abort(7, 7, 7, true, 100) == A::Retired);
+    assert(d4r_input_wait_abort(7, 6, 6, false, 5000001) == A::Timeout);
+    assert(d4r_input_wait_abort(7, 7, 7, false, 5000001) == A::Timeout); // broken wait after input
+    assert(d4r_input_wait_abort(0xffffffffu, 3, 0, false, 100) == A::Discarded);
+    assert(d4r_input_wait_abort(0, 0xffffffffu, 0xffffffffu, false, 100) == A::None);
+    assert(d4r_input_wait_abort(0, 0xffffffffu, 1, false, 100) == A::Discarded);
+    assert(d4r_input_wait_abort(0, 0, 1, false, 100) == A::None);
+    assert(d4r_marker_poll_delay_us(2000, 0) == 200);
+    assert(d4r_marker_poll_delay_us(2000, 1800) == 50);
+    assert(d4r_marker_poll_delay_us(2000, 1850) == 0);
+    assert(d4r_marker_poll_delay_us(2000, 2149) == 0);
+    assert(d4r_marker_poll_delay_us(2000, 2150) == 20);
+    assert(d4r_marker_poll_delay_us(2000, 3750) == 200);
+    assert(d4r_marker_poll_delay_us(0, 5000) == 200); // no warm prediction
+}
+''')
+
+    def test_actual_release_and_drain_helpers_preserve_real_markers_and_ownership(self):
+        shim = (ROOT / 'tools/d4r_nvngx_shim.cpp').read_text()
+        release = function(shim, 'static void release_ngx_input_wait_locked(')
+        complete = function(shim, 'static bool complete_ngx_input_wait(Feature& feature, uint32_t frame, bool abort)\n{')
+        run(r'''
+#include <atomic>
+#include <cassert>
+#include <mutex>
+#include <thread>
+#include <future>
+void logf(const char*, ...) {}
+void MemoryBarrier() { std::atomic_thread_fence(std::memory_order_seq_cst); }
+struct Feature {
+    std::atomic<bool> ngxGpuWait{true}, retiring{false};
+    void* ngxTimeline = nullptr;
+    std::mutex ngxWaitMutex;
+    bool ngxWaitActive = true, ngxWaitCancelled = false;
+    uint32_t ngxWaitFrame = 7;
+    int ngxWaitSlot = 1;
+    struct { void* mapped; } ngxInputSync;
+    unsigned evaluatedFrames = 100;
+};
+uint32_t words[4] = {0, 3, 4, 5}, markers[4] = {6, 3, 4, 5};
+Feature* current;
+int drains = 0, drainResult = 0;
+int sync() {
+    ++drains;
+    assert(words[2] == 7); // drain may only run after the CPU released the blocked queue
+    assert(current->ngxWaitActive); // ticket and input ownership survive the drain
+    return drainResult;
+}
+struct { struct { int(*ctxSynchronize)() = sync; void(*timelineCancel)(void*) = nullptr; } cu; } g;
+''' + release + '\n' + complete + r'''
+int main() {
+    Feature f; f.ngxInputSync.mapped = words; current = &f;
+    assert(complete_ngx_input_wait(f, 8, true)); // another frame cannot release this ticket
+    assert(drains == 0 && words[2] == 4);
+    // An independent thread can release a blocked host submission without invoking HIP.
+    auto blocked = std::async(std::launch::async, [&] {
+        for (;;) {
+            { std::lock_guard<std::mutex> lock(f.ngxWaitMutex); if (words[2] == 7) return; }
+            std::this_thread::yield();
+        }
+    });
+    { std::lock_guard<std::mutex> lock(f.ngxWaitMutex);
+      release_ngx_input_wait_locked(f, "discard", false); }
+    blocked.get();
+    assert(drains == 0 && f.ngxWaitCancelled && f.ngxGpuWait);
+    assert(words[0] == 0 && words[1] == 3 && words[3] == 5);
+    assert(markers[0] == 6 && markers[2] == 4); // never counterfeit game readiness
+    assert(!complete_ngx_input_wait(f, 7, true));
+    assert(drains == 1 && !f.ngxWaitActive && f.evaluatedFrames == 0);
+    assert(complete_ngx_input_wait(f, 7, true) && drains == 1); // idempotent retirement
+    f.ngxWaitActive = true; f.ngxWaitCancelled = false; f.evaluatedFrames = 2;
+    assert(complete_ngx_input_wait(f, 7, false)); // output already drained: no extra sync
+    assert(drains == 1 && !f.ngxWaitActive && f.evaluatedFrames == 2);
+    f.ngxWaitActive = true; f.ngxWaitCancelled = false;
+    { std::lock_guard<std::mutex> lock(f.ngxWaitMutex);
+      release_ngx_input_wait_locked(f, "timeout", true); }
+    assert(!f.ngxGpuWait);
+    drainResult = 700;
+    assert(!complete_ngx_input_wait(f, 7, true));
+    assert(f.retiring && !f.ngxWaitActive && drains == 2);
+}
+''')
+
+    def test_thread_clock_coarse_or_unavailable_reports_na(self):
+        bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
+        clock = function(bridge, 'double WINAPI d4rThreadCpuTimeMs(void)')
+        run(r'''
+#include <time.h>
+#include <cassert>
+#define WINAPI
+long resolutionNs = 1;
+bool available = true;
+int fake_resolution(clockid_t, timespec* out) { out->tv_sec = 0; out->tv_nsec = resolutionNs; return 0; }
+int timestamp(clockid_t, timespec* out) { out->tv_sec = 2; out->tv_nsec = 123456789; return available ? 0 : -1; }
+#define clock_getres fake_resolution
+#define clock_gettime timestamp
+''' + clock + r'''
+int main() {
+    assert(d4rThreadCpuTimeMs() == 2123.456789);
+    resolutionNs = 10000000; assert(d4rThreadCpuTimeMs() == -1);
+    resolutionNs = 1; available = false; assert(d4rThreadCpuTimeMs() == -1);
+}
+''')

tests/test_latency_optimizations.py

@@ -0,0 +1,204 @@
+"""CPU contracts for native completion and packed-output capability gating."""
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def function(source, signature):
+    start = source.index(signature)
+    opening = source.index('{', start)
+    depth, end = 1, opening + 1
+    while depth:
+        depth += (source[end] == '{') - (source[end] == '}')
+        end += 1
+    return source[start:end]
+
+
+def run(source, cpp=False):
+    with tempfile.TemporaryDirectory() as tmp:
+        exe = Path(tmp) / 'check'
+        result = subprocess.run(['c++' if cpp else 'cc', '-x', 'c++' if cpp else 'c',
+                                 '-std=c++20' if cpp else '-std=c11', '-pthread', '-Wall', '-Wextra', '-Werror',
+                                 '-I', str(ROOT / 'tools'), '-o', str(exe), '-'],
+                                input=source, text=True, capture_output=True)
+        if result.returncode: raise AssertionError(result.stderr)
+        subprocess.run([str(exe)], check=True, timeout=5, cwd=tmp)
+
+
+class LatencyContracts(unittest.TestCase):
+    def test_output_membership_filter_keeps_registry_results_exact(self):
+        bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
+        start = bridge.index('enum { OUTPUT_KERNEL_CAPACITY')
+        end = bridge.index('/* 1 when the last output kernel', start)
+        code = bridge[start:end]
+        run(r'''
+#include <stdint.h>
+#include <stddef.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <pthread.h>
+#include <assert.h>
+#define CUfunction void*
+#define R_OK 4
+static const char* fake_getenv(const char* name) { (void)name; return "/objects"; }
+static int fake_access(const char* path, int mode) { (void)path; (void)mode; return 0; }
+#define getenv fake_getenv
+#define access fake_access
+typedef struct { char name[128]; int rgba8; } NativeKernel;
+NativeKernel objects[1] = {{"hiluma_engine_output_candidate", 1}};
+NativeKernel* native_kernels = objects;
+size_t native_kernel_count = 1;
+''' + code + r'''
+int main(void) {
+    void* output = (void*)(uintptr_t)0x5000;
+    note_function_lookup(output, "hiluma_engine_output_candidate"); note_launch(output);
+    assert(last_output_native == 1 && last_output_rgba8 == 1);
+    for (uintptr_t i = 1; i < 1000; ++i) {
+        void* network = (void*)(i * 16);
+        if (network == output) continue;
+        note_launch(network);
+        assert(last_output_native == 1 && last_output_rgba8 == 1); // negatives/collisions change nothing
+    }
+    note_function_lookup(output, "rrlite_downsample_kernel_static_ldr"); note_launch(output);
+    assert(last_output_native == 1 && last_output_rgba8 == 0); // newest lookup wins on handle reuse
+}
+''')
+
+    def test_completed_prefix_callback_gate(self):
+        run(r'''
+#include "d4r_frame_completion.h"
+#include <cassert>
+int main() {
+    D4rFrameCompletion completion;
+    assert(!completion.can_signal(1));
+    completion.admit(1); completion.admit(2); completion.admit(3);
+    assert(completion.can_signal(1) && !completion.can_signal(2));
+    assert(completion.retire(3) == 0); // later discard does not cross unfinished work
+    assert(!completion.can_signal(3));
+    assert(completion.retire(1) == 1 && completion.can_signal(2));
+    assert(completion.retire(2) == 3 && !completion.can_signal(3));
+}
+''', cpp=True)
+
+    def test_actual_native_callback_cancellation_monotonicity_and_detached_cleanup(self):
+        bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
+        start = bridge.index('typedef struct', bridge.index("/* NGX's null-stream completion"))
+        end = bridge.index('} D4rNgxTimeline;', start) + len('} D4rNgxTimeline;')
+        definitions = bridge[start:end]
+        bodies = '\n'.join(function(bridge, sig) for sig in (
+            'int WINAPI d4rNgxTimelineSignal(', 'static void d4r_ngx_completed(',
+            'CUresult WINAPI d4rNgxTimelinePrepare(', 'void WINAPI d4rNgxTimelineCancel(',
+            'CUresult WINAPI d4rNgxTimelineQueue(', 'void WINAPI d4rNgxTimelineDestroy(',
+            'CUresult WINAPI d4rNgxTimelineWait('))
+        run(r'''
+#define _POSIX_C_SOURCE 200809L
+#include <stdint.h>
+#include <stdlib.h>
+#include <pthread.h>
+#include <time.h>
+#include <errno.h>
+#include <assert.h>
+#define WINAPI
+#define CUDA_SUCCESS 0
+#define CUDA_ERROR_NOT_READY 600
+#define CUDA_ERROR_NOT_SUPPORTED 801
+#define CUDA_ERROR_UNKNOWN 999
+#define CUresult int
+typedef struct { int sType; const void* pNext; uint64_t semaphore, value; } D4rVkSemaphoreSignalInfo;
+typedef int (*VK_SIGNAL_SEMAPHORE_FN)(void*, const D4rVkSemaphoreSignalInfo*);
+typedef int (*HIP_LAUNCH_HOST_FN)(void*, void(*)(void*), void*);
+static void* d4r_ngx_stream(void* stream) { return stream; } /* D4R_NGX_STREAM_PRIORITY off */
+void (*callback)(void*); void* argument;
+uint64_t signalled = 0; int signals = 0, launchResult = 0;
+int signal_fake(void* device, const D4rVkSemaphoreSignalInfo* info) {
+    (void)device; assert(info->value > signalled); signalled = info->value; ++signals; return 0;
+}
+static void* complete_later(void* unused) {
+    (void)unused; struct timespec pause = {0, 20000000}; nanosleep(&pause, NULL);
+    callback(argument); return NULL;
+}
+int launch_fake(void* stream, void(*fn)(void*), void* arg) {
+    assert(stream == NULL);
+    if (!launchResult) { callback = fn; argument = arg; }
+    return launchResult;
+}
+''' + definitions + '\n' + bodies + r'''
+int main(void) {
+    D4rNgxTimeline* t = calloc(1, sizeof(*t));
+    assert(pthread_mutex_init(&t->lock, NULL) == 0);
+    pthread_condattr_t attr; assert(pthread_condattr_init(&attr) == 0);
+    assert(pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) == 0);
+    assert(pthread_cond_init(&t->completed, &attr) == 0); pthread_condattr_destroy(&attr);
+    t->signal = signal_fake; t->launch = launch_fake;
+    assert(d4rNgxTimelinePrepare(t) == 0);
+    assert(d4rNgxTimelineQueue(t, 1) == 0);
+    assert(d4rNgxTimelinePrepare(t) == 600 && d4rNgxTimelineQueue(t, 2) == 600);
+    d4rNgxTimelineCancel(t); callback(argument); assert(signals == 0 && t->busy == 0);
+    assert(d4rNgxTimelineWait(t) == 0);
+    assert(d4rNgxTimelineSignal(t, 1) == 0 && signals == 1);
+    assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 2) == 0);
+    callback(argument); assert(signalled == 2 && signals == 2);
+    assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 3) == 0);
+    assert(d4rNgxTimelineSignal(t, 4) == 0); callback(argument);
+    assert(signalled == 4 && signals == 3); // late callback never signals backwards
+    launchResult = 1;
+    assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 5) == 801 && t->busy == 0);
+    launchResult = 0;
+    assert(d4rNgxTimelineQueue(t, 5) == 0);
+    pthread_t worker; assert(pthread_create(&worker, NULL, complete_later, NULL) == 0);
+    assert(d4rNgxTimelineWait(t) == 0); pthread_join(worker, NULL);
+    assert(signalled == 5 && signals == 4);
+    assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 6) == 0);
+    d4rNgxTimelineDestroy(t); callback(argument); // ticket outlives owner, cancellation forbids a signal
+    assert(signals == 4);
+}
+''')
+
+    def test_all_half_values_pack_like_the_reference_unorm_conversion(self):
+        header = (ROOT / 'kernels/tex/tex_common.h').read_text()
+        body = function(header, '__attribute__((device)) static inline uint32_t redirect_unorm(')
+        body = body.replace('__attribute__((device)) ', '')
+        run(r'''
+#include <cstdint>
+#include <cmath>
+#include <cassert>
+''' + body + r'''
+int main() {
+    for (uint32_t bits = 0; bits < 65536; ++bits) {
+        const int exponent = (bits >> 10) & 31, mantissa = bits & 1023;
+        double value = exponent == 0 ? std::ldexp(double(mantissa), -24)
+                     : exponent == 31 ? (mantissa ? NAN : INFINITY)
+                     : std::ldexp(1.0 + double(mantissa) / 1024.0, exponent - 15);
+        if (bits & 0x8000) value = -value;
+        value = value > 0.0 ? value : 0.0;
+        value = value < 1.0 ? value : 1.0;
+        const uint32_t expected = uint32_t(std::floor(value * 255.0 + 0.5));
+        assert(redirect_unorm(uint16_t(bits), 255.0f) == expected);
+    }
+}
+''', cpp=True)
+
+    def test_packed_capability_requires_matching_code_object_bytes(self):
+        bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
+        body = function(bridge, 'static int native_rgba8_verified(')
+        run(r'''
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <assert.h>
+''' + body + r'''
+int main(void) {
+    assert(!native_rgba8_verified("/does/not/exist", "output"));
+    FILE* f = fopen("output.hsaco", "wb"); assert(f != NULL); fputs("abc", f); fclose(f);
+    assert(!native_rgba8_verified(".", "output"));
+    f = fopen("d4r-rgba8.txt", "w"); fputs("output e71fa2190541574b\n", f); fclose(f);
+    assert(native_rgba8_verified(".", "output"));
+    f = fopen("output.hsaco", "ab"); fputc('d', f); fclose(f);
+    assert(!native_rgba8_verified(".", "output"));
+    assert(!native_rgba8_verified(".", "other"));
+}
+''')

tests/test_shim_config.py

@@ -35,9 +35,12 @@ class ShimConfigTests(unittest.TestCase):
             ini = Path(temporary) / 'd4r.ini'
             env = {key: value for key, value in os.environ.items() if not key.startswith('D4R_')}
             cmd = ['python3', str(ROOT / 'scripts/d4r_config.py'), '--config', str(ini)]
-            for value, expected in ((None, '200'), ('auto', '200'), ('200', '200'), ('0', '0'), ('75', '75')):
+            for value, expected in ((None, None), ('auto', None), ('200', '200'), ('0', '0'), ('75', '75')):
                 ini.write_text('[Interop]\n' + (f'MarkerPollUs = {value}\n' if value is not None else ''))
                 result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=True)
+                if expected is None:
+                    self.assertNotIn('export D4R_SHIM_MARKER_POLL_US=', result.stdout)
+                else:
                     self.assertIn(f'export D4R_SHIM_MARKER_POLL_US={expected}\n', result.stdout)
             # An explicit environment variable keeps its existing precedence.
             result = subprocess.run(cmd, env=dict(env, D4R_SHIM_MARKER_POLL_US='35'),

tools/d3d12_dlss_harness.cpp

@@ -1125,9 +1125,9 @@ int main(int argc, char** argv)
         std::fprintf(stderr, "RGB10A2 output cannot be combined with RGBA8 or the quality scene\n");
         return 2;
     }
-    if (rgba8 && (qualityScene || motionScene || jitterScene || replayDir != nullptr))
+    if (rgba8 && (qualityScene || replayDir != nullptr))
     {
-        std::fprintf(stderr, "RGBA8 format probe requires the static synthetic scene\n");
+        std::fprintf(stderr, "RGBA8 format probe supports static, motion and jitter scenes\n");
         return 2;
     }
     const bool temporal = std::getenv("D4R_HARNESS_TEMPORAL") != nullptr || motionScene || jitterScene || qualityScene ||
@@ -1226,6 +1226,8 @@ int main(int argc, char** argv)
         for (size_t index = 0; index < color.size(); ++index)
             color8[index] = static_cast<uint8_t>(std::lround(
                 std::clamp(static_cast<float>(std::bit_cast<_Float16>(color[index])), 0.0f, 1.0f) * 255.0f));
+        if (std::getenv("D4R_HARNESS_RGBA8_RAMP") != nullptr)
+            for (size_t i = 0; i < color8.size(); ++i) color8[i] = static_cast<uint8_t>(i);
         outputInit8.resize(static_cast<size_t>(outWidth) * outHeight * 4);
     }
 
@@ -1421,6 +1423,17 @@ int main(int argc, char** argv)
                                                    : rgb10 ? DXGI_FORMAT_R10G10B10A2_UNORM
                                                            : DXGI_FORMAT_R16G16B16A16_FLOAT,
                                                    D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS);
+    // Moving/jittered RGBA8 fixtures quantize the same canonical half input as the initial upload.
+    // Never upload the half byte layout into an RGBA8 resource: that would make a format test silently
+    // exercise different content instead of checking the conversion/packing route used by the game.
+    const auto update_color = [&] {
+        if (rgba8)
+            for (size_t i = 0; i < color.size(); ++i)
+                color8[i] = static_cast<uint8_t>(std::lround(std::clamp(
+                    static_cast<float>(std::bit_cast<_Float16>(color[i])), 0.0f, 1.0f) * 255.0f));
+        update_texture(colorTexture, rgba8 ? static_cast<const void*>(color8.data())
+                                          : static_cast<const void*>(color.data()), resourceWidth * (rgba8 ? 4 : 8), srv);
+    };
     upload(colorTexture, rgba8 ? static_cast<const void*>(color8.data()) : static_cast<const void*>(color.data()),
            resourceWidth * (rgba8 ? 4 : 8), srv);
     upload(depthTexture, depth.data(), resourceWidth * 4, srv);
@@ -1757,7 +1770,7 @@ int main(int argc, char** argv)
                     color[pixel * 4 + 0] = color[pixel * 4 + 1] = color[pixel * 4 + 2] = half;
                     color[pixel * 4 + 3] = 0x3c00u;
                 }
-            update_texture(colorTexture, color.data(), resourceWidth * 8, srv);
+            update_color();
             if (frame == 1 && (panX != 0 || panY != 0))
             {
                 // Current position -> previous position. Without MVLowRes DLSS
@@ -1800,7 +1813,7 @@ int main(int argc, char** argv)
                 {0.0f, 0.25f, -0.25f, -0.125f, 0.375f, 0.125f, -0.375f, 0.0f};
             const size_t phase = static_cast<size_t>(frame - 1) % jitterX.size();
             render_jittered_background(color, inWidth, inHeight, resourceWidth, jitterX[phase], jitterY[phase]);
-            update_texture(colorTexture, color.data(), resourceWidth * 8, srv);
+            update_color();
             d4r_ngx_set_float(parameters, "Jitter.Offset.X", jitterX[phase]);
             d4r_ngx_set_float(parameters, "Jitter.Offset.Y", jitterY[phase]);
         }
@@ -1823,7 +1836,7 @@ int main(int argc, char** argv)
                     motion[motionPixel * 2 + 0] = 0xc400u; // -4 render pixels: current position -> previous
                     motion[motionPixel * 2 + 1] = 0;
                 }
-            update_texture(colorTexture, color.data(), resourceWidth * 8, srv);
+            update_color();
             update_texture(depthTexture, depth.data(), resourceWidth * 4, srv);
             update_texture(motionTexture, motion_rows(motion.data()), motionRowBytes, srv);
         }
@@ -2054,6 +2067,20 @@ int main(int argc, char** argv)
         LARGE_INTEGER counterStart, counterRecorded, counterEnd, counterFrequency;
         QueryPerformanceFrequency(&counterFrequency);
         QueryPerformanceCounter(&counterStart);
+        // D4R_HARNESS_THRASH_MB=N: copy N MB between two private buffers ahead of the evaluation, as a game's
+        // own rendering does, so DLSS starts with its weights and history evicted from L2 and Infinity Cache
+        // instead of finding them warm from the previous back-to-back frame.
+        static const UINT64 thrashBytes = std::getenv("D4R_HARNESS_THRASH_MB") != nullptr
+            ? std::clamp<UINT64>(std::strtoull(std::getenv("D4R_HARNESS_THRASH_MB"), nullptr, 10), 0, 1024) << 20 : 0;
+        static ID3D12Resource* thrashSource = nullptr;
+        static ID3D12Resource* thrashTarget = nullptr;
+        if (thrashBytes != 0 && thrashSource == nullptr)
+        {
+            thrashSource = create_buffer(D3D12_HEAP_TYPE_DEFAULT, thrashBytes, D3D12_RESOURCE_STATE_COMMON);
+            thrashTarget = create_buffer(D3D12_HEAP_TYPE_DEFAULT, thrashBytes, D3D12_RESOURCE_STATE_COMMON);
+        }
+        if (thrashSource != nullptr && thrashTarget != nullptr)
+            g_list->CopyBufferRegion(thrashTarget, 0, thrashSource, 0, thrashBytes);
         result = evaluate(g_list, feature, parameters, nullptr);
         QueryPerformanceCounter(&counterRecorded);
         if (const char* delay = std::getenv("D4R_HARNESS_DELAY_SUBMIT_MS"))

tools/d4r_frame_completion.h

@@ -11,6 +11,11 @@ class D4rFrameCompletion
 public:
     void admit(uint32_t frame) { pending_.push_back({frame, false}); }
 
+    // A native completion callback can release this value without crossing an unfinished producer
+    // only when it is the first admitted frame still outstanding. Retirement itself stays on the
+    // worker, which also folds any later discarded frames into the completed prefix.
+    bool can_signal(uint32_t frame) const { return !pending_.empty() && pending_.front().frame == frame; }
+
     uint64_t retire(uint32_t frame)
     {
         for (auto& pending : pending_)

tools/d4r_input_wait.h

@@ -0,0 +1,30 @@
+#pragma once
+#include <cstdint>
+
+// The original marker remains the authority for whether a command list ran. A CPU release of the
+// separate, coherent GPU wait word must never make discarded input look ready to the worker.
+enum class D4rInputWaitAbort { None, Retired, Discarded, Timeout };
+
+inline D4rInputWaitAbort d4r_input_wait_abort(uint32_t frame, uint32_t slotMarker, uint32_t aggregate,
+                                            bool retiring, double elapsedUs)
+{
+    if (retiring)
+        return D4rInputWaitAbort::Retired;
+    if (slotMarker != frame && static_cast<int32_t>(aggregate - frame) > 0)
+        return D4rInputWaitAbort::Discarded;
+    // Also covers a stream wait that failed to observe an input word already written by Vulkan.
+    if (elapsedUs > 5000000.0)
+        return D4rInputWaitAbort::Timeout;
+    return D4rInputWaitAbort::None;
+}
+
+inline unsigned d4r_marker_poll_delay_us(double predictedUs, double elapsedUs)
+{
+    const double remaining = predictedUs - 150.0 - elapsedUs;
+    if (remaining > 20.0)
+        return static_cast<unsigned>(remaining < 200.0 ? remaining : 200.0);
+    if (elapsedUs < predictedUs + 150.0)
+        return 0; // yield during the final 150 us and a short late window
+    const double late = 20.0 * (1.0 + (elapsedUs - predictedUs - 150.0) / 160.0);
+    return static_cast<unsigned>(late < 200.0 ? late : 200.0);
+}

tools/d4r_vulkan_staging.h

@@ -16,6 +16,10 @@ struct D4rVkStaging {
     VkDescriptorSetLayout readSet = VK_NULL_HANDLE, writeSet = VK_NULL_HANDLE;
     VkPipelineLayout readLayout = VK_NULL_HANDLE, writeLayout = VK_NULL_HANDLE;
     VkPipeline readPipeline = VK_NULL_HANDLE;
+    // D4R_SHIM_GPU_COMPLETION: one storage buffer of words, push constants {index, value, limit}
+    VkDescriptorSetLayout waitSet = VK_NULL_HANDLE;
+    VkPipelineLayout waitLayout = VK_NULL_HANDLE;
+    VkPipeline waitPipeline = VK_NULL_HANDLE;
     struct OutputPipeline { VkFormat format; VkPipeline pipeline; } outputs[5] = {};
     PFN_vkCreateSampler createSampler = nullptr;
     PFN_vkDestroySampler destroySampler = nullptr;
@@ -81,7 +85,7 @@ struct D4rVkStaging {
             set.bindingCount = 2; set.pBindings = bindings;
             auto& setLayout = output ? writeSet : readSet;
             if (createSetLayout(d, &set, nullptr, &setLayout) != VK_SUCCESS) return false;
-            VkPushConstantRange range = {VK_SHADER_STAGE_COMPUTE_BIT, 0, output ? 8u : 12u};
+            VkPushConstantRange range = {VK_SHADER_STAGE_COMPUTE_BIT, 0, output ? 8u : 16u};
             VkPipelineLayoutCreateInfo layout = {VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO};
             layout.setLayoutCount = 1; layout.pSetLayouts = &setLayout;
             layout.pushConstantRangeCount = 1; layout.pPushConstantRanges = ⦥
@@ -89,6 +93,44 @@ struct D4rVkStaging {
         }
         return pipeline(d4r_vk_read, sizeof(d4r_vk_read), readLayout, readPipeline);
     }
+    // Created on first use; false leaves GPU completion off.
+    bool wait_ready() {
+        if (waitPipeline) return true;
+        VkDescriptorSetLayoutBinding binding = {0, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, nullptr};
+        VkDescriptorSetLayoutCreateInfo set = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO};
+        set.bindingCount = 1; set.pBindings = &binding;
+        if (!waitSet && createSetLayout(device, &set, nullptr, &waitSet) != VK_SUCCESS) return false;
+        VkPushConstantRange range = {VK_SHADER_STAGE_COMPUTE_BIT, 0, 12};
+        VkPipelineLayoutCreateInfo layout = {VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO};
+        layout.setLayoutCount = 1; layout.pSetLayouts = &waitSet;
+        layout.pushConstantRangeCount = 1; layout.pPushConstantRanges = ⦥
+        if (!waitLayout && createLayout(device, &layout, nullptr, &waitLayout) != VK_SUCCESS) return false;
+        return pipeline(d4r_vk_wait, sizeof(d4r_vk_wait), waitLayout, waitPipeline);
+    }
+    bool wait_set(D4rVkStagingSet& result, VkBuffer buffer, VkDeviceSize bytes) {
+        if (!wait_ready()) return false;
+        VkDescriptorPoolSize size = {VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1};
+        VkDescriptorPoolCreateInfo pool = {VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO};
+        pool.maxSets = 1; pool.poolSizeCount = 1; pool.pPoolSizes = &size;
+        if (!result.pool && createPool(device, &pool, nullptr, &result.pool) != VK_SUCCESS) return false;
+        VkDescriptorSetAllocateInfo alloc = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO};
+        alloc.descriptorPool = result.pool; alloc.descriptorSetCount = 1; alloc.pSetLayouts = &waitSet;
+        if (!result.set && allocateSets(device, &alloc, &result.set) != VK_SUCCESS) return false;
+        VkDescriptorBufferInfo storage = {buffer, 0, bytes};
+        VkWriteDescriptorSet write = {VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET};
+        write.dstSet = result.set; write.dstBinding = 0; write.descriptorCount = 1;
+        write.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER; write.pBufferInfo = &storage;
+        updateSets(device, 1, &write, 0, nullptr);
+        result.pipeline = waitPipeline;
+        return true;
+    }
+    void record_wait(VkCommandBuffer cmd, const D4rVkStagingSet& set, uint32_t index, uint32_t value, uint32_t limit) {
+        bindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, waitPipeline);
+        bindSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, waitLayout, 0, 1, &set.set, 0, nullptr);
+        const uint32_t constants[] = {index, value, limit};
+        push(cmd, waitLayout, VK_SHADER_STAGE_COMPUTE_BIT, 0, 12, constants);
+        dispatch(cmd, 1, 1, 1);
+    }
     VkPipeline output_pipeline(VkFormat format) {
         for (auto& item : outputs) if (item.pipeline && item.format == format) return item.pipeline;
         const uint32_t* code = nullptr; size_t bytes = 0;
@@ -135,12 +177,12 @@ struct D4rVkStaging {
         updateSets(device, 2, writes, 0, nullptr);
         return true;
     }
-    void record(VkCommandBuffer cmd, const D4rVkStagingSet& set, uint32_t width, uint32_t height, uint32_t inputMode = 0) {
+    void record(VkCommandBuffer cmd, const D4rVkStagingSet& set, uint32_t width, uint32_t height, uint32_t inputMode = 0, uint32_t stride = 0) {
         const auto layout = set.output ? writeLayout : readLayout;
         bindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, set.pipeline);
         bindSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, layout, 0, 1, &set.set, 0, nullptr);
-        const uint32_t dimensions[] = {width, height, inputMode};
+        const uint32_t dimensions[] = {width, height, inputMode, stride ? stride : width};
-        push(cmd, layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, set.output ? 8 : 12, dimensions);
+        push(cmd, layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, set.output ? 8 : 16, dimensions);
         dispatch(cmd, (width+7)/8, (height+7)/8, 1);
     }
     void free(D4rVkStagingSet& set) {
@@ -149,6 +191,9 @@ struct D4rVkStaging {
     }
     void destroy() {
         if (readPipeline) destroyPipeline(device, readPipeline, nullptr);
+        if (waitPipeline) destroyPipeline(device, waitPipeline, nullptr);
+        if (waitLayout) destroyLayout(device, waitLayout, nullptr);
+        if (waitSet) destroySetLayout(device, waitSet, nullptr);
         for (auto& item : outputs) if (item.pipeline) destroyPipeline(device, item.pipeline, nullptr);
         if (readLayout) destroyLayout(device, readLayout, nullptr);
         if (writeLayout) destroyLayout(device, writeLayout, nullptr);

tools/wine_nvcuda_bridge.c

@@ -11,6 +11,7 @@
 #include <string.h>
 #include <sys/stat.h>
 #include <unistd.h>
+#include <time.h>
 #include "d4r_native_selection.h"
 #include "d4r_motion_dilation.h"
 #ifdef D4R_MOTION_KERNELS
@@ -366,11 +367,41 @@ typedef struct
     char name[128];
     uint64_t hash;
     int linked;
+    int rgba8;
 } NativeKernel;
 
 static NativeKernel* native_kernels;
 static size_t native_kernel_count;
 static char native_source[1024];
+
+/* Packed stores are a code-object capability, not a property of the kernel's name. A legacy native
+   kernel ignores the R4 redirect tag. The sidecar must therefore name the exact bytes installed
+   alongside the PTX compatibility manifest; the per-process cache retains those same bytes. */
+static int native_rgba8_verified(const char* directory, const char* name)
+{
+    char path[1400], row[256], entry[128];
+    unsigned long long expected = 0;
+    snprintf(path, sizeof(path), "%s/d4r-rgba8.txt", directory);
+    FILE* manifest = fopen(path, "r");
+    if (manifest == NULL) return 0;
+    int found = 0;
+    while (fgets(row, sizeof(row), manifest) != NULL)
+        if (sscanf(row, "%127s %llx", entry, &expected) == 2 && strcmp(entry, name) == 0)
+        { found = 1; break; }
+    fclose(manifest);
+    if (!found) return 0;
+    snprintf(path, sizeof(path), "%s/%s.hsaco", directory, name);
+    FILE* object = fopen(path, "rb");
+    if (object == NULL) return 0;
+    uint64_t hash = UINT64_C(14695981039346656037);
+    unsigned char bytes[4096];
+    size_t count;
+    while ((count = fread(bytes, 1, sizeof(bytes), object)) != 0)
+        for (size_t i = 0; i < count; ++i) hash = (hash ^ bytes[i]) * UINT64_C(1099511628211);
+    const int valid = !ferror(object) && hash == (uint64_t)expected;
+    fclose(object);
+    return valid;
+}
 static char native_served[1024];
 static pthread_mutex_t native_lock = PTHREAD_MUTEX_INITIALIZER;
 
@@ -506,6 +537,7 @@ static void prepare_native_kernels(const char* cache_home)
         snprintf(kernel->name, sizeof(kernel->name), "%s", name);
         kernel->hash = hash;
         kernel->linked = 0;
+        kernel->rgba8 = native_rgba8_verified(source, name);
     }
     fclose(manifest);
     if (!d4r_native_swin_encoders_enabled())
@@ -733,9 +765,22 @@ static struct
 {
     CUfunction function;
     int native;
+    int rgba8;
 } output_kernels[OUTPUT_KERNEL_CAPACITY]; /* the latest lookups, newest last */
 static pthread_mutex_t output_kernel_lock = PTHREAD_MUTEX_INITIALIZER;
 static int last_output_native = -1;
+static int last_output_rgba8;
+/* Most NGX launches are network/input kernels. A monotonic membership filter lets those bypass the
+   output registry's lock and scan. Collisions only take the old lookup path; a bit is never cleared,
+   so function-handle reuse, feature recreation and concurrent lookups cannot hide an output kernel. */
+static uint64_t output_kernel_bits[16];
+
+static unsigned output_kernel_bit(CUfunction function)
+{
+    uint64_t value = (uint64_t)(uintptr_t)function >> 4;
+    value ^= value >> 17;
+    return (unsigned)((value * UINT64_C(0x9e3779b97f4a7c15)) >> 54);
+}
 
 static void note_function_lookup(CUfunction function, const char* name)
 {
@@ -755,20 +800,34 @@ static void note_function_lookup(CUfunction function, const char* name)
     memmove(output_kernels, output_kernels + 1, sizeof(output_kernels) - sizeof(output_kernels[0]));
     output_kernels[OUTPUT_KERNEL_CAPACITY - 1].function = function;
     output_kernels[OUTPUT_KERNEL_CAPACITY - 1].native = native;
+    int rgba8 = 0;
+    for (size_t i = 0; i < native_kernel_count; ++i)
+        if (strcmp(native_kernels[i].name, name) == 0) { rgba8 = native_kernels[i].rgba8; break; }
+    output_kernels[OUTPUT_KERNEL_CAPACITY - 1].rgba8 = native && rgba8;
     pthread_mutex_unlock(&output_kernel_lock);
+    const unsigned bit = output_kernel_bit(function);
+    __atomic_fetch_or(&output_kernel_bits[bit >> 6], UINT64_C(1) << (bit & 63), __ATOMIC_RELEASE);
 }
 
-static void note_launch(CUfunction function)
+/* Returns 1 + native + 2 * rgba8 when `function` is an output kernel, 0 otherwise. */
+static int note_launch(CUfunction function)
 {
+    const unsigned bit = output_kernel_bit(function);
+    if ((__atomic_load_n(&output_kernel_bits[bit >> 6], __ATOMIC_ACQUIRE) & (UINT64_C(1) << (bit & 63))) == 0)
+        return 0;
+    int kind = 0;
     pthread_mutex_lock(&output_kernel_lock);
     /* newest first, so a recycled handle resolves with its latest lookup */
     for (int i = OUTPUT_KERNEL_CAPACITY - 1; i >= 0 && output_kernels[i].function != NULL; --i)
         if (output_kernels[i].function == function)
         {
             __atomic_store_n(&last_output_native, output_kernels[i].native, __ATOMIC_RELAXED);
+            __atomic_store_n(&last_output_rgba8, output_kernels[i].rgba8, __ATOMIC_RELAXED);
+            kind = 1 + output_kernels[i].native + 2 * output_kernels[i].rgba8;
             break;
         }
     pthread_mutex_unlock(&output_kernel_lock);
+    return kind;
 }
 
 /* 1 when the last output kernel launched was d4r's native one, 0 when it was ZLUDA's compile of
@@ -778,6 +837,11 @@ int WINAPI d4rOutputKernelNative(void)
     return __atomic_load_n(&last_output_native, __ATOMIC_RELAXED);
 }
 
+int WINAPI d4rOutputKernelRgba8(void)
+{
+    return __atomic_load_n(&last_output_rgba8, __ATOMIC_RELAXED);
+}
+
 static void load_zluda(void)
 {
     /* Explicit experimental backend. Never silently mix module/context/array
@@ -964,7 +1028,7 @@ CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters
     D4rMotionDilationParams copy = *parameters;
     void* arguments[] = {&copy};
     const uint64_t pixels = (uint64_t)copy.motion_width * copy.motion_height;
-    return launch(function, (unsigned)((pixels + 127) / 128), 1, 1, 128, 1, 1, 0, NULL, arguments, NULL);
+    return launch(function, (unsigned)((pixels + 127) / 128), 1, 1, 128, 1, 1, 0, d4r_ngx_stream(NULL), arguments, NULL);
 #else
     return CUDA_ERROR_NOT_SUPPORTED;
 #endif
@@ -1127,6 +1191,16 @@ static void ensure_context(void)
     pthread_once(&context_once, create_default_zluda_context);
 }
 
+/* D4R_NGX_STREAM_PRIORITY=high|normal (default off): the null-stream work of NGX, the shim and this bridge
+   (kernels, async copies and memsets, events, stream waits, input waits, completion callbacks) moves from
+   HIP's null stream to one blocking HIP stream. "high" creates it at the device's greatest priority, so its
+   HSA queue (and the hardware queue it maps to) dispatches ahead of normal-priority queues; "normal" creates
+   it at the default priority, the control that isolates the stream change itself. Every caller passes NULL
+   and gets the same stream, so the order of the work is unchanged; being blocking, the stream also stays
+   ordered with ZLUDA's own synchronous null-stream operations. Off: NULL stays HIP's null stream. */
+static CUstream d4r_ngx_stream(CUstream stream);
+static void* hip_symbol(const char* name);
+
 /* Opt-in stage instrumentation. With D4R_CUDA_LAUNCH_STATS=1 every
    cuLaunchKernel is followed by a context synchronize and a summary of each
    device allocation, surface object and texture object named in the packed
@@ -1402,15 +1476,17 @@ static CUresult write_redirect_tail(CUsurfObject object, CUdeviceptr pointer, ui
     static unsigned int next;
     unsigned char* tail = ring[next++ % 256];
     // dword 21: 0x5232 ('R2', RGBA16F texels) << 16 | pitch / 8 (0 = off); dwords 22-23: pointer.
-    // Row pitches are multiples of 8, so bit 0 of `pitch` selects 'R3': R10G10B10A2_UNORM texels.
+    // Row pitches are multiples of 8: bit 0 selects R3 (RGB10A2), bit 1 selects R4 (RGBA8).
-    const uint32_t word = pitch != 0 ? ((pitch & 1u) != 0 ? 0x52330000u : 0x52320000u) | ((pitch >> 3) & 0xffffu) : 0u;
+    const uint32_t word = pitch != 0 ? ((pitch & 2u) != 0 ? 0x52340000u :
+                                      (pitch & 1u) != 0 ? 0x52330000u : 0x52320000u) |
+                                      ((pitch >> 3) & 0xffffu) : 0u;
     memset(tail, 0, 16);
     memcpy(tail, &word, 4);
     memcpy(tail + 4, &pointer, 8);
     CUMEMCPYHTODASYNC_FN function = (CUMEMCPYHTODASYNC_FN)find_zluda_symbol("cuMemcpyHtoDAsync_v2");
     if (function == NULL)
         return CUDA_ERROR_NOT_SUPPORTED;
-    CUresult result = function((CUdeviceptr)object + 84, tail, 12, NULL);
+    CUresult result = function((CUdeviceptr)object + 84, tail, 12, d4r_ngx_stream(NULL));
     if (result == CUDA_SUCCESS && !async)
     {
         CUCTX_SYNCHRONIZE_FN sync = (CUCTX_SYNCHRONIZE_FN)find_zluda_symbol("cuCtxSynchronize");
@@ -2431,6 +2507,33 @@ CUresult WINAPI cuModuleGetFunction(CUfunction* function_out, CUmodule module, c
     return result;
 }
 
+/* D4R_SHIM_GPU_COMPLETION, early form: the shim arms a completion word before NGX's evaluate. When the output
+   kernel that follows is d4r's native one (which writes the game-visible buffer directly; RGBA8-capable when
+   the frame needs the packed store), the word is queued right behind that kernel, ahead of NGX's trailing
+   bookkeeping kernels, which the game never reads. Any other output kernel leaves it to the shim's own write
+   after the output copy. Only the CUDA worker evaluates, so the arm state needs no lock. */
+static CUdeviceptr ngx_completion_pointer;
+static uint32_t ngx_completion_value;
+static int ngx_completion_rgba8, ngx_completion_fired;
+
+CUresult WINAPI d4rNgxArmCompletion(CUdeviceptr pointer, uint32_t value, int rgba8)
+{
+    ngx_completion_pointer = pointer;
+    ngx_completion_value = value;
+    ngx_completion_rgba8 = rgba8;
+    ngx_completion_fired = 0;
+    return CUDA_SUCCESS;
+}
+
+/* 1 when the armed word was queued behind the native output kernel; disarms. */
+int WINAPI d4rNgxCompletionFired(void)
+{
+    const int fired = ngx_completion_fired;
+    ngx_completion_pointer = 0;
+    ngx_completion_fired = 0;
+    return fired;
+}
+
 CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x, unsigned int grid_y,
                                unsigned int grid_z, unsigned int block_x, unsigned int block_y,
                                unsigned int block_z, unsigned int shared_bytes, CUstream stream,
@@ -2439,13 +2542,11 @@ CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x,
     static CULAUNCHKERNEL_FN function;
     if (function == NULL)
         function = (CULAUNCHKERNEL_FN)find_zluda_symbol("cuLaunchKernel");
-    unsigned int sequence = 0;
+    stream = d4r_ngx_stream(stream);
-    pthread_mutex_lock(&instrumentation_lock);
+    const unsigned int sequence = __atomic_add_fetch(&launch_sequence, 1, __ATOMIC_RELAXED);
-    sequence = ++launch_sequence;
-    pthread_mutex_unlock(&instrumentation_lock);
     replay_dump_launch(sequence, function_handle, grid_x, grid_y, grid_z, block_x, block_y, block_z,
                        shared_bytes, extra);
-    note_launch(function_handle);
+    const int output_kind = note_launch(function_handle);
     PendingKernelProfile* profile = function != NULL
         ? begin_kernel_profile(sequence, function_handle, stream) : NULL;
     CUfunction native_function = microcuda_k_lookup(function_handle);
@@ -2468,6 +2569,17 @@ CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x,
         tracef("microcuda-k prep stats runs=%llu skips=%llu", (unsigned long long)prep, (unsigned long long)skips);
     }
     end_kernel_profile(profile, result, stream);
+    if (result == CUDA_SUCCESS && ngx_completion_pointer != 0 && output_kind != 0)
+    {
+        const int native = ((output_kind - 1) & 1) != 0, rgba8 = (output_kind - 1) >= 2;
+        typedef int (*WRITE_FN)(void*, void*, uint32_t, unsigned int);
+        static WRITE_FN write;
+        if (write == NULL)
+            write = (WRITE_FN)hip_symbol("hipStreamWriteValue32");
+        if (native && (!ngx_completion_rgba8 || rgba8) && write != NULL)
+            ngx_completion_fired = write(stream, (void*)(uintptr_t)ngx_completion_pointer, ngx_completion_value, 0) == 0;
+        ngx_completion_pointer = 0; /* one output kernel per evaluate: never fire on a later one */
+    }
     if (result != CUDA_SUCCESS || trace_verbose())
     {
         tracef("cuLaunchKernel[%u] %s", sequence, find_function_name(function_handle));
@@ -2569,6 +2681,7 @@ CUresult WINAPI cuMemcpyHtoDAsync(CUdeviceptr destination, const void* source, s
 {
     CUMEMCPYHTODASYNC_FN function = (CUMEMCPYHTODASYNC_FN)find_zluda_symbol("cuMemcpyHtoDAsync_v2");
     if (microcuda_k_write) microcuda_k_write(destination, bytes);
+    stream = d4r_ngx_stream(stream);
     CUresult result = function != NULL ? function(destination, source, bytes, stream) : missing("cuMemcpyHtoDAsync_v2");
     TRACE_CALL(result, "cuMemcpyHtoDAsync->v2 dst=0x%llx src=%p bytes=%zu stream=%p result=%d",
            (unsigned long long)destination, source, bytes, stream, result);
@@ -2906,7 +3019,6 @@ CUresult WINAPI cuTexObjectGetResourceDesc(void* descriptor, CUtexObject object)
     return result;
 }
 
-
 /* Direct pass-throughs for driver APIs whose arguments are all integers or
    pointers, so the Windows x64 call forwards unchanged to ZLUDA (SysV). Unversioned
    names resolve to the versions the CUDA 12 driver returns for them. */
@@ -2935,8 +3047,8 @@ static int d4r_forward_is_teardown(const char* target)
 D4R_FORWARD(cuEventCreate, cuEventCreate, 2, (D4rArg a0, D4rArg a1), (a0, a1))
 D4R_FORWARD(cuEventDestroy, cuEventDestroy_v2, 1, (D4rArg a0), (a0))
 D4R_FORWARD(cuEventDestroy_v2, cuEventDestroy_v2, 1, (D4rArg a0), (a0))
-D4R_FORWARD(cuEventRecord, cuEventRecord, 2, (D4rArg a0, D4rArg a1), (a0, a1))
+D4R_FORWARD(cuEventRecord, cuEventRecord, 2, (D4rArg a0, D4rArg a1), (a0, (D4rArg)d4r_ngx_stream((CUstream)a1)))
-D4R_FORWARD(cuEventRecordWithFlags, cuEventRecordWithFlags, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
+D4R_FORWARD(cuEventRecordWithFlags, cuEventRecordWithFlags, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, (D4rArg)d4r_ngx_stream((CUstream)a1), a2))
 CUresult WINAPI cuEventSynchronize(D4rArg a0)
 {
     if (before_teardown())
@@ -2995,9 +3107,9 @@ D4R_FORWARD(cuStreamCreate, cuStreamCreate, 2, (D4rArg a0, D4rArg a1), (a0, a1))
 D4R_FORWARD(cuStreamCreateWithPriority, cuStreamCreateWithPriority, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
 D4R_FORWARD(cuStreamDestroy, cuStreamDestroy_v2, 1, (D4rArg a0), (a0))
 D4R_FORWARD(cuStreamDestroy_v2, cuStreamDestroy_v2, 1, (D4rArg a0), (a0))
-D4R_FORWARD(cuStreamSynchronize, cuStreamSynchronize, 1, (D4rArg a0), (a0))
+D4R_FORWARD(cuStreamSynchronize, cuStreamSynchronize, 1, (D4rArg a0), ((D4rArg)d4r_ngx_stream((CUstream)a0)))
-D4R_FORWARD(cuStreamQuery, cuStreamQuery, 1, (D4rArg a0), (a0))
+D4R_FORWARD(cuStreamQuery, cuStreamQuery, 1, (D4rArg a0), ((D4rArg)d4r_ngx_stream((CUstream)a0)))
-D4R_FORWARD(cuStreamWaitEvent, cuStreamWaitEvent, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
+D4R_FORWARD(cuStreamWaitEvent, cuStreamWaitEvent, 3, (D4rArg a0, D4rArg a1, D4rArg a2), ((D4rArg)d4r_ngx_stream((CUstream)a0), a1, a2))
 D4R_FORWARD(cuStreamGetFlags, cuStreamGetFlags, 2, (D4rArg a0, D4rArg a1), (a0, a1))
 D4R_FORWARD(cuStreamGetPriority, cuStreamGetPriority, 2, (D4rArg a0, D4rArg a1), (a0, a1))
 D4R_FORWARD(cuCtxGetCurrent, cuCtxGetCurrent, 1, (D4rArg a0), (a0))
@@ -3014,14 +3126,14 @@ D4R_FORWARD(cuMemGetInfo_v2, cuMemGetInfo_v2, 2, (D4rArg a0, D4rArg a1), (a0, a1
 D4R_FORWARD(cuMemcpyHtoD, cuMemcpyHtoD_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
 D4R_FORWARD(cuMemcpyHtoD_v2, cuMemcpyHtoD_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
 D4R_FORWARD(cuMemcpyDtoD, cuMemcpyDtoD_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
-D4R_FORWARD(cuMemcpyDtoDAsync, cuMemcpyDtoDAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
+D4R_FORWARD(cuMemcpyDtoDAsync, cuMemcpyDtoDAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
-D4R_FORWARD(cuMemcpyDtoHAsync, cuMemcpyDtoHAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
+D4R_FORWARD(cuMemcpyDtoHAsync, cuMemcpyDtoHAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
 D4R_FORWARD(cuMemsetD8, cuMemsetD8_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
 D4R_FORWARD(cuMemsetD16, cuMemsetD16_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
 D4R_FORWARD(cuMemsetD32, cuMemsetD32_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
-D4R_FORWARD(cuMemsetD8Async, cuMemsetD8Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
+D4R_FORWARD(cuMemsetD8Async, cuMemsetD8Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
-D4R_FORWARD(cuMemsetD16Async, cuMemsetD16Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
+D4R_FORWARD(cuMemsetD16Async, cuMemsetD16Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
-D4R_FORWARD(cuMemsetD32Async, cuMemsetD32Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
+D4R_FORWARD(cuMemsetD32Async, cuMemsetD32Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
 D4R_FORWARD(cuMemsetD2D8, cuMemsetD2D8_v2, 5, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3, D4rArg a4), (a0, a1, a2, a3, a4))
 D4R_FORWARD(cuMemsetD2D32, cuMemsetD2D32_v2, 5, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3, D4rArg a4), (a0, a1, a2, a3, a4))
 D4R_FORWARD(cuFuncSetAttribute, cuFuncSetAttribute, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
@@ -3116,6 +3228,59 @@ static void* hip_symbol(const char* name)
     return hip != NULL ? dlsym(hip, name) : NULL;
 }
 
+static CUstream ngx_priority_stream;
+static pthread_once_t ngx_priority_once = PTHREAD_ONCE_INIT;
+
+static void create_ngx_priority_stream(void)
+{
+    const char* setting = getenv("D4R_NGX_STREAM_PRIORITY");
+    const int high = setting != NULL && strcmp(setting, "high") == 0;
+    if (!high && (setting == NULL || strcmp(setting, "normal") != 0))
+        return;
+    typedef int (*STREAM_PRIORITY_RANGE_FN)(int*, int*);
+    typedef int (*STREAM_CREATE_PRIORITY_FN)(void**, unsigned int, int);
+    STREAM_PRIORITY_RANGE_FN range = (STREAM_PRIORITY_RANGE_FN)hip_symbol("hipDeviceGetStreamPriorityRange");
+    STREAM_CREATE_PRIORITY_FN create = (STREAM_CREATE_PRIORITY_FN)hip_symbol("hipStreamCreateWithPriority");
+    int least = 0, greatest = 0;
+    ensure_context();
+    if (context_setup_result != CUDA_SUCCESS || range == NULL || create == NULL || range(&least, &greatest) != 0)
+    {
+        tracef("D4R_NGX_STREAM_PRIORITY=%s: HIP stream priorities unavailable; NGX stays on the null stream", setting);
+        return;
+    }
+    void* stream = NULL;
+    const int priority = high ? greatest : 0;
+    const int result = create(&stream, 0 /* hipStreamDefault: blocking */, priority);
+    tracef("D4R_NGX_STREAM_PRIORITY=%s: NGX stream %p priority %d (range %d..%d) result=%d", setting, stream,
+           priority, least, greatest, result);
+    if (result == 0)
+        ngx_priority_stream = (CUstream)stream;
+}
+
+static CUstream d4r_ngx_stream(CUstream stream)
+{
+    if (stream != NULL)
+        return stream;
+    pthread_once(&ngx_priority_once, create_ngx_priority_stream);
+    return ngx_priority_stream; /* NULL unless D4R_NGX_STREAM_PRIORITY created one */
+}
+
+CUresult WINAPI d4rNgxStreamPriority(int* priority)
+{
+    /* For the shim's log: 1 when NGX runs on the substituted stream, with its HIP priority. */
+    typedef int (*STREAM_GET_PRIORITY_FN)(void*, int*);
+    CUstream stream = d4r_ngx_stream(NULL);
+    STREAM_GET_PRIORITY_FN get = (STREAM_GET_PRIORITY_FN)hip_symbol("hipStreamGetPriority");
+    if (priority == NULL)
+        return CUDA_ERROR_INVALID_VALUE;
+    *priority = 0;
+    if (stream == NULL)
+        return CUDA_ERROR_NOT_SUPPORTED;
+    if (get != NULL)
+        get(stream, priority);
+    return CUDA_SUCCESS;
+}
+
 CUresult WINAPI d4rImportVulkanMemory(void* client_device, uint64_t client_memory, uint64_t bytes, CUdeviceptr* pointer,
                                       void** memory)
 {
@@ -3282,6 +3447,146 @@ typedef int (*VK_SIGNAL_SEMAPHORE_FN)(void*, const D4rVkSemaphoreSignalInfo*);
 typedef int (*VK_GET_SEMAPHORE_COUNTER_FN)(void*, uint64_t, uint64_t*);
 typedef int (*HIP_LAUNCH_HOST_FN)(void*, void (*)(void*), void*);
 
+/* NGX's null-stream completion, independent of Wine scheduling/event polling. CPU cancellations
+   and native callbacks share one timeline lock so a late callback never signals a lower value.
+   No HIP or Wine APIs may run from the callback. The shim queues this only for the completed-prefix
+   head, after every output write; resource retirement waits for its native completion condition. */
+typedef struct
+{
+    void* device;
+    uint64_t semaphore, value, pending;
+    VK_SIGNAL_SEMAPHORE_FN signal;
+    HIP_LAUNCH_HOST_FN launch;
+    pthread_mutex_t lock;
+    pthread_cond_t completed;
+    int busy, cancelled, detached;
+} D4rNgxTimeline;
+
+int WINAPI d4rNgxTimelineSignal(void* handle, uint64_t value)
+{
+    D4rNgxTimeline* timeline = handle;
+    pthread_mutex_lock(&timeline->lock);
+    int result = 0;
+    if (value > timeline->value)
+    {
+        D4rVkSemaphoreSignalInfo info = {1000207005, NULL, timeline->semaphore, value};
+        result = timeline->signal(timeline->device, &info);
+        if (result == 0) timeline->value = value;
+    }
+    pthread_mutex_unlock(&timeline->lock);
+    return result;
+}
+
+static void d4r_ngx_completed(void* data)
+{
+    D4rNgxTimeline* timeline = data;
+    pthread_mutex_lock(&timeline->lock);
+    if (!__atomic_load_n(&timeline->cancelled, __ATOMIC_ACQUIRE) && timeline->pending > timeline->value)
+    {
+        D4rVkSemaphoreSignalInfo info = {1000207005, NULL, timeline->semaphore, timeline->pending};
+        if (timeline->signal(timeline->device, &info) == 0) timeline->value = timeline->pending;
+        /* A failed signal leaves the worker's normal retirement to retry. */
+    }
+    timeline->busy = 0;
+    pthread_cond_signal(&timeline->completed);
+    const int detached = timeline->detached;
+    pthread_mutex_unlock(&timeline->lock);
+    if (detached) { pthread_cond_destroy(&timeline->completed); pthread_mutex_destroy(&timeline->lock); free(timeline); }
+}
+
+CUresult WINAPI d4rNgxTimelineCreate(void* client_device, uint64_t client_semaphore, void** handle)
+{
+    if (client_device == NULL || client_semaphore == 0 || handle == NULL) return CUDA_ERROR_INVALID_VALUE;
+    const uint64_t* device_object = (const uint64_t*)(uintptr_t)((const uint64_t*)client_device)[1];
+    const uint64_t* semaphore_object = (const uint64_t*)(uintptr_t)client_semaphore;
+    if (device_object == NULL || device_object[1] != (uint64_t)(uintptr_t)client_device ||
+        semaphore_object[1] != client_semaphore) return CUDA_ERROR_NOT_SUPPORTED;
+    void* vulkan = dlopen("libvulkan.so.1", RTLD_NOW | RTLD_NOLOAD);
+    VK_GET_DEVICE_PROC_ADDR_FN get = vulkan != NULL ? dlsym(vulkan, "vkGetDeviceProcAddr") : NULL;
+    HIP_LAUNCH_HOST_FN launch = (HIP_LAUNCH_HOST_FN)hip_symbol("hipLaunchHostFunc");
+    if (get == NULL || launch == NULL) return CUDA_ERROR_NOT_SUPPORTED;
+    void* device = (void*)(uintptr_t)device_object[0];
+    VK_SIGNAL_SEMAPHORE_FN signal = (VK_SIGNAL_SEMAPHORE_FN)get(device, "vkSignalSemaphore");
+    VK_GET_SEMAPHORE_COUNTER_FN counter = (VK_GET_SEMAPHORE_COUNTER_FN)get(device, "vkGetSemaphoreCounterValue");
+    uint64_t value = 0;
+    if (signal == NULL || counter == NULL || counter(device, semaphore_object[0], &value) != 0)
+        return CUDA_ERROR_NOT_SUPPORTED;
+    D4rNgxTimeline* timeline = calloc(1, sizeof(*timeline));
+    if (timeline == NULL) return CUDA_ERROR_OUT_OF_MEMORY;
+    if (pthread_mutex_init(&timeline->lock, NULL) != 0) { free(timeline); return CUDA_ERROR_UNKNOWN; }
+    pthread_condattr_t attr;
+    if (pthread_condattr_init(&attr) != 0) { pthread_mutex_destroy(&timeline->lock); free(timeline); return CUDA_ERROR_UNKNOWN; }
+    const int condition = pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) != 0 ||
+                          pthread_cond_init(&timeline->completed, &attr) != 0;
+    pthread_condattr_destroy(&attr);
+    if (condition) { pthread_mutex_destroy(&timeline->lock); free(timeline); return CUDA_ERROR_UNKNOWN; }
+    timeline->device = device; timeline->semaphore = semaphore_object[0]; timeline->value = value;
+    timeline->signal = signal; timeline->launch = launch; *handle = timeline;
+    return CUDA_SUCCESS;
+}
+
+CUresult WINAPI d4rNgxTimelinePrepare(void* handle)
+{
+    D4rNgxTimeline* timeline = handle;
+    pthread_mutex_lock(&timeline->lock);
+    const int busy = timeline->busy;
+    if (!busy) __atomic_store_n(&timeline->cancelled, 0, __ATOMIC_RELEASE);
+    pthread_mutex_unlock(&timeline->lock);
+    return busy ? CUDA_ERROR_NOT_READY : CUDA_SUCCESS;
+}
+
+void WINAPI d4rNgxTimelineCancel(void* handle)
+{
+    /* Called by the independent input watchdog before its coherent release store: no queue calls/locks. */
+    __atomic_store_n(&((D4rNgxTimeline*)handle)->cancelled, 1, __ATOMIC_RELEASE);
+}
+
+CUresult WINAPI d4rNgxTimelineQueue(void* handle, uint64_t value)
+{
+    D4rNgxTimeline* timeline = handle;
+    pthread_mutex_lock(&timeline->lock);
+    if (timeline->busy) { pthread_mutex_unlock(&timeline->lock); return CUDA_ERROR_NOT_READY; }
+    timeline->pending = value; timeline->busy = 1;
+    pthread_mutex_unlock(&timeline->lock);
+    const int result = timeline->launch(d4r_ngx_stream(NULL), d4r_ngx_completed, timeline);
+    if (result != 0)
+    {
+        pthread_mutex_lock(&timeline->lock); timeline->busy = 0; pthread_mutex_unlock(&timeline->lock);
+    }
+    return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_NOT_SUPPORTED;
+}
+
+void WINAPI d4rNgxTimelineDestroy(void* handle)
+{
+    D4rNgxTimeline* timeline = handle;
+    pthread_mutex_lock(&timeline->lock);
+    __atomic_store_n(&timeline->cancelled, 1, __ATOMIC_RELEASE);
+    const int busy = timeline->busy;
+    timeline->detached = busy;
+    pthread_mutex_unlock(&timeline->lock);
+    /* Normally the native condition drained the callback. If a device error prevented that drain, retain
+       only this self-owned ticket until its cancelled callback returns, never a pointer into Feature. */
+    if (!busy) { pthread_cond_destroy(&timeline->completed); pthread_mutex_destroy(&timeline->lock); free(timeline); }
+}
+
+CUresult WINAPI d4rNgxTimelineWait(void* handle)
+{
+    D4rNgxTimeline* timeline = handle;
+    struct timespec deadline;
+    if (clock_gettime(CLOCK_MONOTONIC, &deadline) != 0) return CUDA_ERROR_UNKNOWN;
+    deadline.tv_sec += 5;
+    pthread_mutex_lock(&timeline->lock);
+    int result = 0;
+    while (timeline->busy && result == 0)
+        result = pthread_cond_timedwait(&timeline->completed, &timeline->lock, &deadline);
+    const int busy = timeline->busy;
+    pthread_mutex_unlock(&timeline->lock);
+    /* The callback runs after every output write on the null stream. Its condition is sufficient to
+       retire those resources, without polling/recording another GPU event. A timeout keeps the ticket
+       owned; the caller cancels, disables this mode and falls back to its normal output synchronization. */
+    return !busy ? CUDA_SUCCESS : result == ETIMEDOUT ? CUDA_ERROR_NOT_READY : CUDA_ERROR_UNKNOWN;
+}
+
 typedef struct D4rEngineNet
 {
     int k_preset;  /* D4RKHIP1: the token grid comes from the output dimensions and the cycle has eight graphs */
@@ -3333,7 +3638,6 @@ typedef int (*HIP_CAPTURE_BEGIN_FN)(void*, int);
 typedef int (*HIP_CAPTURE_END_FN)(void*, void**);
 typedef int (*HIP_GRAPH_INSTANTIATE_FN)(void**, void*, void**, char*, size_t);
 
-
 CUresult WINAPI d4rEngineNetReleaseInput(void* handle, uint32_t value)
 {
     D4rEngineNet* net = (D4rEngineNet*)handle;
@@ -4175,7 +4479,7 @@ CUresult WINAPI d4rMemcpy2DAsync(const D4rMemcpy2D* copy, CUstream stream)
     D4rMemcpy2D hip = *copy;
     hip.srcMemoryType = (unsigned int)src;
     hip.dstMemoryType = (unsigned int)dst;
-    const int result = function(&hip, NULL);
+    const int result = function(&hip, d4r_ngx_stream(NULL));
     if (result != 0 || trace_verbose())
         tracef("d4rMemcpy2DAsync %zux%zu result=%d", copy->WidthInBytes, copy->Height, result);
     return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_INVALID_VALUE;
@@ -4194,12 +4498,68 @@ CUresult WINAPI d4rStreamWaitValue32(CUdeviceptr pointer, uint32_t value)
         function = (HIP_STREAM_WAIT_VALUE32_FN)hip_symbol("hipStreamWaitValue32");
     if (function == NULL)
         return CUDA_ERROR_NOT_SUPPORTED;
-    const int result = function(NULL, (void*)(uintptr_t)pointer, value, 0 /* hipStreamWaitValueGte */, 0xffffffffu);
+    const int result = function(d4r_ngx_stream(NULL), (void*)(uintptr_t)pointer, value, 0 /* hipStreamWaitValueGte */,
+                                0xffffffffu);
     if (result != 0 || trace_verbose())
         tracef("d4rStreamWaitValue32 %p >= %u result=%d", (void*)(uintptr_t)pointer, value, result);
     return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_INVALID_VALUE;
 }
 
+/* Exact per-slot tags, rather than the engine's monotonic Gte word: an old or later frame must never
+   satisfy NGX's wait, including across u32 wrap. ZLUDA uses HIP's null stream for null CUstream kernel
+   launches; d4rMemcpy2DAsync uses that same stream. This export therefore orders both input copies and
+   NGX kernels without adding an unsupported CUDA driver entry point to ZLUDA. The caller owns a coherent
+   CPU mapping and arms its watchdog before entering here, so a blocked HIP call can still be released. */
+CUresult WINAPI d4rStreamWaitValue32Eq(CUdeviceptr pointer, uint32_t value)
+{
+    typedef int (*GET_ATTRIBUTE_FN)(int*, int, int);
+    typedef int (*GET_DEVICE_FN)(int*);
+    ensure_context();
+    if (context_setup_result != CUDA_SUCCESS)
+        return context_setup_result;
+    HIP_STREAM_WAIT_VALUE32_FN wait = (HIP_STREAM_WAIT_VALUE32_FN)hip_symbol("hipStreamWaitValue32");
+    GET_ATTRIBUTE_FN attribute = (GET_ATTRIBUTE_FN)hip_symbol("hipDeviceGetAttribute");
+    GET_DEVICE_FN device = (GET_DEVICE_FN)hip_symbol("hipGetDevice");
+    int current = 0, supported = 0;
+    if (pointer == 0 || wait == NULL || attribute == NULL || device == NULL || device(&current) != 0 ||
+        attribute(&supported, 10013 /* hipDeviceAttributeCanUseStreamWaitValue */, current) != 0 || !supported)
+        return CUDA_ERROR_NOT_SUPPORTED;
+    const int result = wait(d4r_ngx_stream(NULL), (void*)(uintptr_t)pointer, value, 1 /* hipStreamWaitValueEq */,
+                            0xffffffffu);
+    if (result != 0 || trace_verbose())
+        tracef("d4rStreamWaitValue32Eq %p == %u result=%d", (void*)(uintptr_t)pointer, value, result);
+    return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_NOT_SUPPORTED; /* failure: no wait queued */
+}
+
+double WINAPI d4rThreadCpuTimeMs(void)
+{
+    struct timespec resolution, now;
+    if (clock_getres(CLOCK_THREAD_CPUTIME_ID, &resolution) != 0 || resolution.tv_sec != 0 ||
+        resolution.tv_nsec > 100000 /* 100 us: too coarse for these sub-ms submission/wait costs */ ||
+        clock_gettime(CLOCK_THREAD_CPUTIME_ID, &now) != 0)
+        return -1.0;
+    return (double)now.tv_sec * 1000.0 + (double)now.tv_nsec / 1000000.0;
+}
+
+/* D4R_SHIM_GPU_COMPLETION: queues a 32-bit write of `value` to `pointer` on the stream NGX runs on, behind the
+   frame's output writes. The game's queue polls that word (kernels/support/vulkan_wait.comp) instead of waiting
+   for a CPU-signalled timeline value, so DLSS's completion reaches the game without a host round trip. */
+CUresult WINAPI d4rNgxWriteValue32(CUdeviceptr pointer, uint32_t value)
+{
+    static HIP_STREAM_WRITE_VALUE32_FN write;
+    ensure_context();
+    if (context_setup_result != CUDA_SUCCESS)
+        return context_setup_result;
+    if (write == NULL)
+        write = (HIP_STREAM_WRITE_VALUE32_FN)hip_symbol("hipStreamWriteValue32");
+    if (write == NULL || pointer == 0)
+        return CUDA_ERROR_NOT_SUPPORTED;
+    const int result = write(d4r_ngx_stream(NULL), (void*)(uintptr_t)pointer, value, 0);
+    if (result != 0 || trace_verbose())
+        tracef("d4rNgxWriteValue32 %p = %u result=%d", (void*)(uintptr_t)pointer, value, result);
+    return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_NOT_SUPPORTED;
+}
+
 CUresult WINAPI d4rWriteValue32(CUdeviceptr pointer, uint32_t value)
 {
     static HIP_STREAM_WRITE_VALUE32_FN write;

tools/wine_nvcuda_bridge.spec

@@ -104,10 +104,24 @@
 @ stdcall d4rSetShutdownHook(ptr) d4rSetShutdownHook
 @ stdcall d4rEventSynchronize(ptr) d4rEventSynchronize
 @ stdcall d4rStreamWaitValue32(int64 long) d4rStreamWaitValue32
+@ stdcall d4rStreamWaitValue32Eq(int64 long) d4rStreamWaitValue32Eq
+@ stdcall d4rNgxStreamPriority(ptr) d4rNgxStreamPriority
+@ stdcall d4rNgxWriteValue32(int64 long) d4rNgxWriteValue32
+@ stdcall d4rNgxArmCompletion(int64 long long) d4rNgxArmCompletion
+@ stdcall d4rNgxCompletionFired() d4rNgxCompletionFired
+@ stdcall d4rThreadCpuTimeMs() d4rThreadCpuTimeMs
 @ stdcall d4rWriteValue32(int64 long) d4rWriteValue32
 @ stdcall d4rSetArrayRedirect(ptr int64 long) d4rSetArrayRedirect
 @ stdcall d4rRegisterLinearTexture(int64 int64 int64 long long) d4rRegisterLinearTexture
 @ stdcall d4rSetEnv(str str long) d4rSetEnv
 @ stdcall d4rLoadError() d4rLoadError
 @ stdcall d4rOutputKernelNative() d4rOutputKernelNative
+@ stdcall d4rOutputKernelRgba8() d4rOutputKernelRgba8
+@ stdcall d4rNgxTimelineCreate(ptr int64 ptr) d4rNgxTimelineCreate
+@ stdcall d4rNgxTimelinePrepare(ptr) d4rNgxTimelinePrepare
+@ stdcall d4rNgxTimelineCancel(ptr) d4rNgxTimelineCancel
+@ stdcall d4rNgxTimelineQueue(ptr int64) d4rNgxTimelineQueue
+@ stdcall d4rNgxTimelineSignal(ptr int64) d4rNgxTimelineSignal
+@ stdcall d4rNgxTimelineDestroy(ptr) d4rNgxTimelineDestroy
+@ stdcall d4rNgxTimelineWait(ptr) d4rNgxTimelineWait
 @ stdcall d4rDilateMotionVectors(ptr) d4rDilateMotionVectors