shim: adaptive marker polling, opt-in GPU-side input wait/completion and RGBA8 output
- [Interop] MarkerPollUs = auto is the new default: the input-marker wait learns when each feature's inputs become ready, sleeps in <=200 us chunks, yields near the prediction and backs off when late (d4r_input_wait.h); a number keeps fixed pacing (0 yields). d4r_config.py no longer exports 200 when the key is unset - D4R_SHIM_GPU_INPUT_WAIT=1 (experimental, split VRAM, age 0): NGX's null stream waits on a coherent Vulkan word imported into HIP (d4rStreamWaitValue32Eq) instead of CPU polling. The original readback marker stays the authority; a CPU watchdog releases discarded lists and timeouts. Denoiser, verify, capture and dilation frames keep the CPU wait - D4R_SHIM_GPU_COMPLETION=1 (needs GPU_INPUT_WAIT): the bridge writes a completion word right after the native output kernel (d4rNgxWriteValue32), and the game's queue polls it with a bounded compute wait (kernels/support/vulkan_wait.comp), replacing the split list's CPU timeline signal. frame_retired writes the same word on every failure path - D4R_SHIM_NATIVE_COMPLETION=1: a null-stream callback in the bridge signals the split timeline (d4rNgxTimeline*); D4rFrameCompletion::can_signal only releases the oldest outstanding frame - RGBA8 output: tex_common.h adds an 'R4' redirect tag, so K's native output kernel stores packed RGBA8 directly (D4R_SHIM_OUTPUT_RGBA8=1, checked against the kernel manifest by d4rOutputKernelRgba8). Host packing handles RGBA8 too. patches/vkd3d-proton/0004 (VKD3D_D4R_LINEAR_RGBA8=WxH) makes such outputs linear and exportable so the result can be written in place - D4R_SHIM_FUSE_RGBA8_INPUT=1: one Vulkan pass converts and copies RGBA8 color input (vulkan_read.comp mode 5 matches the blit's toward-zero halves, plus a row stride) - profiling: thread CPU time from the Unix bridge (Wine's GetThreadTimes ticks at 10 ms); game-queue D3D12 timestamps (vk_input/vk_dlss_wait/vk_output/vk_total); gpu_eval split into input_gpu_wait and dlss_gpu; D4R_PROFILE_GPU_EVENTS, _TIMESTAMPS, _LOG and _CPU_TIME=0 switch off individual parts for bisection - bridge: D4R_NGX_STREAM_PRIORITY=high|normal runs NGX on a priority stream (off by default; no gain measured) - release_feature: a feature already marked retiring by a failed drain can still be released normally - harness: RGBA8 with motion and jitter scenes (re-quantized each frame), D4R_HARNESS_RGBA8_RAMP, D4R_HARNESS_THRASH_MB to evict L2/Infinity Cache before each evaluation - kernels/native: in_hoist.py and split_output_paths.py (opt-in scheduling experiments); scripts/build_k_ldr_output.sh builds K's LDR output with an optional D4R_LDR_OUTPUT_MAX_VGPR cap - tests: test_input_wait.py and test_latency_optimizations.py; MarkerPollUs and frame-completion tests updated
This commit is contained in:
ayo committed
1 parent
f0b55a850a
commit
162bbd77c8
22 files changed
+1981
-101
No files matched your search
@@ -97,18 +97,19 @@ VramInterop = true
|
||||
InputSync = true
|
||||
; Faster hand-offs between the game and DLSS (same frame, same result). Need FrameAge = 0 and VRAM
|
||||
; interop, and a runtime built from this checkout (scripts/install_d4r_runtime.sh).
|
||||
; Input readiness is checked with CPU sleeps; recorded-but-discarded lists never queue GPU waits.
|
||||
; Input readiness uses adaptive CPU polling by default. D4R_SHIM_GPU_INPUT_WAIT=1 enables experimental
|
||||
; null-stream GPU waits for same-frame Super Sampling; diagnostics and host input reads retain CPU waits.
|
||||
; LinearInputs: DLSS reads the game's inputs in place once the input marker is ready
|
||||
; DirectOutput: the result is written straight into the game-side buffer (preset K, and only while
|
||||
; NGX runs the native output kernel; drops NVIDIA's "DLSS SDK" watermark)
|
||||
; ElideNgxSync: removes NGX's internal CPU waits between its GPU stages
|
||||
; EvalSync = false: no CPU wait between DLSS and the output copy
|
||||
; MarkerPollUs: how often (microseconds) the CPU checks for the game's inputs
|
||||
; MarkerPollUs: auto learns readiness and yields near it; N fixes the poll interval in microseconds
|
||||
LinearInputs = true
|
||||
DirectOutput = true
|
||||
ElideNgxSync = true
|
||||
EvalSync = false
|
||||
MarkerPollUs = 200
|
||||
MarkerPollUs = auto
|
||||
|
||||
[Engine]
|
||||
; Experimental. Runs DLSS 4 and 4.5 (models K, L and M) on d4r's own Vulkan inference engine instead of NVIDIA's
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Issue independent texture fetches of the translated hiluma input kernel earlier (opt-in experiment).
|
||||
|
||||
usage: in_hoist.py KERNEL.hip ANCHOR_REG HANDLE_REG [HANDLE_REG ...] > OUT.hip
|
||||
|
||||
The input kernel (ptx2hip --ifconv --sink + in_lds2reg.py) is one straight-line body. Its 3x3 color neighbourhood
|
||||
fetches depend only on the thread's pixel position and kernel parameters, yet the compiler issues them after the
|
||||
depth -> motion -> history chain, adding serial memory round trips to a latency-bound kernel. This pass moves every
|
||||
fetch block `{ ::f4 t_ = tex_fetch(rdX, ...); ... }` whose handle register is one of HANDLE_REG's copies, together
|
||||
with the backward slice of statements computing its operands, to right after the statement that assigns
|
||||
ANCHOR_REG, followed by a scheduling barrier (instruction order only). Nothing else moves and no statement
|
||||
changes, so every value is computed exactly as before.
|
||||
|
||||
Refuses unless: every register the moved statements assign is assigned exactly once in the kernel; every register
|
||||
they read is assigned before the anchor (or by another moved statement, earlier in order); there is no memory store
|
||||
and no control flow (label/goto/return) between the anchor and the last moved statement.
|
||||
"""
|
||||
import re
|
||||
import sys
|
||||
|
||||
REG = re.compile(r'\b(?:rd|rs|r|f|p|shv\d+_|dpv\d+_)\d+\b|\bl\d+_\w+\b')
|
||||
ASSIGN = re.compile(r'^\s*([A-Za-z_]\w*)\s*=(?!=)')
|
||||
|
||||
|
||||
def stmts(body_lines):
|
||||
"""group lines into statements: fetch blocks '{ ... }' become one statement"""
|
||||
out, i = [], 0
|
||||
while i < len(body_lines):
|
||||
line = body_lines[i]
|
||||
if line.strip().startswith('{ ::f4 t_ ='):
|
||||
j = i
|
||||
while body_lines[j].strip() != '}':
|
||||
j += 1
|
||||
out.append(body_lines[i:j + 1])
|
||||
i = j + 1
|
||||
else:
|
||||
out.append([line])
|
||||
i += 1
|
||||
return out
|
||||
|
||||
|
||||
def defs_uses(st):
|
||||
text = ''.join(st)
|
||||
if text.strip().startswith('{ ::f4 t_ ='):
|
||||
defs = set(re.findall(r'^\s*([A-Za-z_]\w*) = t_\[\d\];', text, re.M))
|
||||
call = text.split(';', 1)[0].replace('::f4', '')
|
||||
return defs, set(REG.findall(call))
|
||||
m = ASSIGN.match(text)
|
||||
if m:
|
||||
lhs = m.group(1)
|
||||
rhs = text[m.end():]
|
||||
return {lhs}, set(REG.findall(rhs))
|
||||
return set(), set(REG.findall(text))
|
||||
|
||||
|
||||
def main():
|
||||
src = open(sys.argv[1]).read().split('\n')
|
||||
anchor, handles = sys.argv[2], set(sys.argv[3:])
|
||||
start = next(i for i, l in enumerate(src) if l.startswith('extern "C" __attribute__((global))')) + 2
|
||||
end = max(i for i, l in enumerate(src) if l.strip() == '}')
|
||||
S = stmts([l + '\n' for l in src[start:end]])
|
||||
info = [defs_uses(s) for s in S]
|
||||
assigned = {}
|
||||
for k, (d, _) in enumerate(info):
|
||||
for r in d:
|
||||
assigned.setdefault(r, []).append(k)
|
||||
# handle copies: rdY = (uint64_t)(rdX) chains from the given handle registers
|
||||
alias = set(handles)
|
||||
for k, s in enumerate(S):
|
||||
m = re.match(r'^\s*(rd\d+) = \(uint64_t\)\((rd\d+)\);$', s[0])
|
||||
if m and m.group(2) in alias:
|
||||
alias.add(m.group(1))
|
||||
ia = [k for k, (d, _) in enumerate(info) if anchor in d]
|
||||
if len(ia) != 1:
|
||||
raise SystemExit(f'anchor {anchor} assigned {len(ia)} times')
|
||||
ia = ia[0]
|
||||
fetches = [k for k, s in enumerate(S) if s[0].strip().startswith('{ ::f4 t_ = tex_fetch(')
|
||||
and re.search(r'tex_fetch\((rd\d+)', s[0]).group(1) in alias and k > ia]
|
||||
if not fetches:
|
||||
raise SystemExit('no fetch to move')
|
||||
need, move = set(), set(fetches)
|
||||
for k in fetches:
|
||||
need |= info[k][1]
|
||||
while need:
|
||||
r = need.pop()
|
||||
ks = assigned.get(r, [])
|
||||
if not ks:
|
||||
continue # parameters / builtins
|
||||
if len(ks) != 1:
|
||||
raise SystemExit(f'register {r} is assigned {len(ks)} times')
|
||||
k = ks[0]
|
||||
if k <= ia or k in move:
|
||||
continue
|
||||
move.add(k)
|
||||
need |= info[k][1]
|
||||
last = max(move)
|
||||
for k in range(ia + 1, last + 1):
|
||||
text = ''.join(S[k])
|
||||
if re.search(r'^\s*\*\(|\bgoto\b|\breturn\b|^L__BB', text, re.M):
|
||||
raise SystemExit(f'store or control flow between anchor and moved code: {text.strip()[:80]}')
|
||||
for k in sorted(move):
|
||||
for r in info[k][0]:
|
||||
if len(assigned[r]) != 1:
|
||||
raise SystemExit(f'moved register {r} assigned more than once')
|
||||
for r in info[k][1]:
|
||||
ks = assigned.get(r, [])
|
||||
if ks and not (ks[0] <= ia or ks[0] in move and ks[0] < k):
|
||||
raise SystemExit(f'moved statement reads {r} defined later')
|
||||
# a scheduling barrier keeps the compiler from sinking the fetches back to their uses (order only)
|
||||
barrier = [' __builtin_amdgcn_sched_barrier(0); // in_hoist: fetches above are issued here\n']
|
||||
body = ''.join(''.join(S[k]) for k in range(0, ia + 1)) + ''.join(''.join(S[k]) for k in sorted(move)) + \
|
||||
''.join(barrier) + ''.join(''.join(S[k]) for k in range(ia + 1, len(S)) if k not in move)
|
||||
sys.stderr.write(f'in_hoist: moved {len(move)} statements ({len(fetches)} fetches) after {anchor}\n')
|
||||
sys.stdout.write('\n'.join(src[:start]) + '\n' + body + '\n'.join(src[end:]))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Split a translated hiluma output kernel into its two per-block paths (opt-in experiment).
|
||||
|
||||
usage: split_output_paths.py KERNEL.hip > KERNEL_split.hip
|
||||
|
||||
NVIDIA's hiluma_engine_output_* kernels pick one of two code paths per block: path B (an LDS tile of the
|
||||
block's render footprint) when the footprint is under 24 texels in both axes, which holds for every block at
|
||||
Quality .. Ultra Performance, else path A (direct fetches, DLAA). The choice depends only on the block index and
|
||||
kernel parameters, so it is uniform per block, and blocks share nothing. Path A needs more registers than path B
|
||||
(LDR mvlo: 101 vs 88 VGPRs), and one kernel carries the larger allocation for every block.
|
||||
|
||||
This pass emits the same translated body twice:
|
||||
NAME runs path B and returns where path A would start;
|
||||
NAME_post1 runs path A and returns where path B would start.
|
||||
The ZLUDA/MicroCUDA native hook launches NAME_post1 right after NAME on the same stream with the same parameters
|
||||
and grid (d4r_post1_grid_x/_y = 0), with the 16x16 blocks the PTX hard-codes (tid.y << 4). Each block executes
|
||||
exactly the instructions it executed before; only the block order across the two paths changes, which no output
|
||||
depends on (blocks never read what other blocks write in the same launch).
|
||||
|
||||
The pass refuses anything but the expected entry structure:
|
||||
pA = ((int32_t)rX < (int32_t)24); pB = ((int32_t)rY < (int32_t)24); pC = (pA && pB);
|
||||
if (pC) goto LB;
|
||||
goto LA;
|
||||
"""
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def split(source):
|
||||
head, sep, main = source.partition('extern "C" __attribute__((global))')
|
||||
if not sep or 'extern "C" __attribute__((global))' in main:
|
||||
raise SystemExit('expected exactly one kernel')
|
||||
m = re.match(r'\s*__attribute__\(\(amdgpu_flat_work_group_size\(1, 256\)\)\) void (\w+)\((\w+)\)', main)
|
||||
if not m:
|
||||
raise SystemExit('unexpected kernel signature')
|
||||
name = m.group(1)
|
||||
pattern = re.compile(r'^ (p\d+) = \(\(int32_t\)r\d+ < \(int32_t\)24\);\n'
|
||||
r' (p\d+) = \(\(int32_t\)r\d+ < \(int32_t\)24\);\n'
|
||||
r' (p\d+) = \((p\d+) && (p\d+)\);\n'
|
||||
r' if \((p\d+)\) goto (L__BB0_\d+);\n'
|
||||
r' goto (L__BB0_\d+);\n', re.M)
|
||||
found = list(pattern.finditer(main))
|
||||
if len(found) != 1:
|
||||
raise SystemExit(f'path selection found {len(found)} times, expected once')
|
||||
g = found[0]
|
||||
pa, pb, pc, x, y, pc2, path_b, path_a = g.groups()
|
||||
if {x, y} != {pa, pb} or pc != pc2:
|
||||
raise SystemExit('path selection predicate has an unexpected form')
|
||||
# the predicate must not be reassigned anywhere (it is the block's path choice)
|
||||
if len(re.findall(rf'^ {pc} = ', main, re.M)) != 1:
|
||||
raise SystemExit(f'{pc} is assigned more than once')
|
||||
sel = g.group(0)
|
||||
lines = sel.splitlines(keepends=True)
|
||||
main_b = main.replace(sel, ''.join(lines[:4]) + ' return; // path A blocks: NAME_post1\n', 1)
|
||||
main_a = main.replace(sel, ''.join(lines[:3]) + f' if ({pc}) return; // path B blocks: NAME\n'
|
||||
+ f' goto {path_a};\n', 1)
|
||||
main_a = main_a.replace(f' void {name}(', f' void {name}_post1(', 1)
|
||||
globals_ = ('\n// d4r native hook: launch NAME_post1 after NAME with the launch grid and the PTX\'s 16x16 blocks\n'
|
||||
'extern "C" __attribute__((device, used)) uint32_t d4r_post1_grid_x = 0;\n'
|
||||
'extern "C" __attribute__((device, used)) uint32_t d4r_post1_grid_y = 0;\n'
|
||||
'extern "C" __attribute__((device, used)) uint32_t d4r_post1_block_x = 16;\n'
|
||||
'extern "C" __attribute__((device, used)) uint32_t d4r_post1_block_y = 16;\n'
|
||||
'extern "C" __attribute__((device, used)) uint32_t d4r_post1_block_z = 1;\n')
|
||||
banner = '// split_output_paths.py: NAME = path B only, NAME_post1 = path A only (per-block choice unchanged)\n'
|
||||
return (banner + head + globals_ + sep + main_b + '\n' + sep + main_a, path_b, path_a)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
text, b, a = split(open(sys.argv[1]).read())
|
||||
sys.stderr.write(f'split: path B {b}, path A {a}\n')
|
||||
sys.stdout.write(text)
|
||||
@@ -2,7 +2,7 @@
|
||||
layout(local_size_x=8, local_size_y=8) in;
|
||||
layout(set=0, binding=0) uniform sampler2D sourceImage;
|
||||
layout(std430, set=0, binding=1) writeonly buffer Pixels { uint pixels[]; };
|
||||
layout(push_constant) uniform Size { uint width; uint height; uint mode; } size;
|
||||
layout(push_constant) uniform Size { uint width; uint height; uint mode; uint stride; } size;
|
||||
// Match the CPU canonical conversion, including round-to-even and subnormals.
|
||||
uint halfBits(float value) {
|
||||
uint bits = floatBitsToUint(value);
|
||||
@@ -26,14 +26,24 @@ uint halfBits(float value) {
|
||||
if (remainder > halfway || (remainder == halfway && (m & 1u) != 0u)) ++result;
|
||||
return result;
|
||||
}
|
||||
// Mode 5 is only for RGBA8 UNORM: values are zero or normal positive halves. Preserve
|
||||
// the blit's toward-zero conversion; other modes keep the canonical round-to-even path.
|
||||
uint halfRtz(float value) {
|
||||
uint b = floatBitsToUint(value);
|
||||
if (value == 0.0) return 0u;
|
||||
return (((b >> 23) - 112u) << 10) | ((b & 0x7fffffu) >> 13);
|
||||
}
|
||||
void main() {
|
||||
uvec2 p = gl_GlobalInvocationID.xy;
|
||||
if (p.x >= size.width || p.y >= size.height) return;
|
||||
ivec2 extent = textureSize(sourceImage, 0);
|
||||
uint index = p.y * size.width + p.x;
|
||||
uint index = p.y * size.stride + p.x;
|
||||
vec4 value = all(lessThan(ivec2(p), extent))
|
||||
? texelFetch(sourceImage, ivec2(p), 0) : vec4(0.0);
|
||||
if (size.mode == 1u) { // canonical RGBA16F color
|
||||
if (size.mode == 5u) {
|
||||
pixels[index * 2u] = halfRtz(value.x) | (halfRtz(value.y) << 16);
|
||||
pixels[index * 2u + 1u] = halfRtz(value.z) | (halfRtz(value.w) << 16);
|
||||
} else if (size.mode == 1u) { // canonical RGBA16F color
|
||||
pixels[index * 2u] = halfBits(value.x) | (halfBits(value.y) << 16);
|
||||
pixels[index * 2u + 1u] = halfBits(value.z) | (halfBits(value.w) << 16);
|
||||
} else if (size.mode == 3u) { // canonical RG16F motion
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
#version 450
|
||||
// D4R_SHIM_GPU_COMPLETION: holds the game's queue until DLSS has written this frame's completion word (HIP
|
||||
// hipStreamWriteValue32 after the output, or the CPU on every failure path). Bounded: after `limit` polls the
|
||||
// shader returns regardless, so a lost write can never hang the GPU (the frame then shows its previous output).
|
||||
layout(local_size_x = 1) in;
|
||||
layout(std430, set = 0, binding = 0) coherent volatile readonly buffer Words { uint words[]; };
|
||||
layout(push_constant) uniform Wait { uint index; uint value; uint limit; } wait;
|
||||
void main() {
|
||||
for (uint poll = 0u; poll < wait.limit; ++poll)
|
||||
if (words[wait.index] == wait.value)
|
||||
return;
|
||||
}
|
||||
@@ -371,15 +371,19 @@ extern "C" __attribute__((device)) int __ockl_image_width_2D(tsharp_t* image);
|
||||
extern "C" __attribute__((device)) int __ockl_image_height_2D(tsharp_t* image);
|
||||
__attribute__((device)) static inline uint32_t redirect_pitch(uint64_t surface)
|
||||
{
|
||||
// 'R2' (RGBA16F texels) or 'R3' (R10G10B10A2_UNORM texels) tag in the high half (other handles may have
|
||||
// 'R2' (RGBA16F), 'R3' (RGB10A2) or 'R4' (RGBA8) tag in the high half (other handles may have
|
||||
// data there), row pitch / 8 in the low half
|
||||
const uint32_t word = ((const __attribute__((address_space(4))) uint32_t*)surface)[21];
|
||||
return (word >> 16) == 0x5232u || (word >> 16) == 0x5233u ? (word & 0xffffu) << 3 : 0u;
|
||||
return (word >> 16) >= 0x5232u && (word >> 16) <= 0x5234u ? (word & 0xffffu) << 3 : 0u;
|
||||
}
|
||||
__attribute__((device)) static inline bool redirect_rgb10a2(uint64_t surface)
|
||||
{
|
||||
return (((const __attribute__((address_space(4))) uint32_t*)surface)[21] >> 16) == 0x5233u;
|
||||
}
|
||||
__attribute__((device)) static inline bool redirect_rgba8(uint64_t surface)
|
||||
{
|
||||
return (((const __attribute__((address_space(4))) uint32_t*)surface)[21] >> 16) == 0x5234u;
|
||||
}
|
||||
// One channel as the game-side path produces it: the f32 value rounded toward zero to a half (the RGBA16F
|
||||
// store), then converted to UNORM by a blit (NaN and negatives 0, clamped at 1, rounded to nearest).
|
||||
__attribute__((device)) static inline uint32_t redirect_unorm(uint16_t half, float scale)
|
||||
@@ -422,6 +426,10 @@ DEV __attribute__((always_inline)) void d4r_sust_p_v4b32(uint64_t surface, int32
|
||||
*D4R_REDIRECT_PTR(uint32_t, redirect_row(surface, y) + 4 * x) =
|
||||
redirect_unorm((uint16_t)lo, 1023.0f) | redirect_unorm((uint16_t)(lo >> 16), 1023.0f) << 10 |
|
||||
redirect_unorm((uint16_t)hi, 1023.0f) << 20 | redirect_unorm((uint16_t)(hi >> 16), 3.0f) << 30;
|
||||
else if (redirect_rgba8(surface))
|
||||
*D4R_REDIRECT_PTR(uint32_t, redirect_row(surface, y) + 4 * x) =
|
||||
redirect_unorm((uint16_t)lo, 255.0f) | redirect_unorm((uint16_t)(lo >> 16), 255.0f) << 8 |
|
||||
redirect_unorm((uint16_t)hi, 255.0f) << 16 | redirect_unorm((uint16_t)(hi >> 16), 255.0f) << 24;
|
||||
else
|
||||
*D4R_REDIRECT_PTR(uint2_t, redirect_row(surface, y) + 8 * x) = (uint2_t){lo, hi};
|
||||
}
|
||||
@@ -526,9 +534,15 @@ DEV __attribute__((always_inline)) void d4r_sust_b32(uint64_t surface, int32_t x
|
||||
tsharp_t* image = (tsharp_t*)surface;
|
||||
if (redirect_pitch(surface) != 0)
|
||||
{
|
||||
// Raw RGBA16F stores carry two half channels at each four-byte offset. Packing those channels
|
||||
// separately keeps the original store ownership (no read-modify-write or cross-thread race).
|
||||
if (redirect_rgba8(surface) && x >= 0 && (x & 3) == 0 && redirect_inside(surface, x >> 3, y))
|
||||
*D4R_REDIRECT_PTR(uint16_t, redirect_row(surface, y) + (x >> 1)) =
|
||||
(uint16_t)(redirect_unorm((uint16_t)data, 255.0f) |
|
||||
redirect_unorm((uint16_t)(data >> 16), 255.0f) << 8);
|
||||
// raw bits at byte offset x (RGBA16F rows are 8 bytes per pixel); the shim only asks for packed
|
||||
// RGB10A2 texels from kernels without raw stores
|
||||
if (!redirect_rgb10a2(surface) && x >= 0 && (x & 3) == 0 && redirect_inside(surface, x >> 3, y))
|
||||
// RGB10A2 texels from kernels without raw stores; RGBA8 preserves each half-pair store
|
||||
if (!redirect_rgb10a2(surface) && !redirect_rgba8(surface) && x >= 0 && (x & 3) == 0 && redirect_inside(surface, x >> 3, y))
|
||||
*D4R_REDIRECT_PTR(uint32_t, redirect_row(surface, y) + x) = data;
|
||||
return;
|
||||
}
|
||||
|
||||
+3
-3
@@ -62,10 +62,10 @@ NativeFp8 = on
|
||||
; Requires render- or display-resolution, unjittered motion vectors and same-frame linear VRAM inputs. Restart to change.
|
||||
; Unsupported features keep original vectors; larger radii may affect disocclusion edges.
|
||||
MotionVectorDilation = 0
|
||||
; Keep inputs and output in video memory. Input and output readiness waits use CPU sleeps.
|
||||
; Keep inputs and output in video memory. Readiness uses adaptive CPU polling by default.
|
||||
VramInterop = true
|
||||
; Microseconds between checks for the game's input marker (default 200; 0 yields).
|
||||
MarkerPollUs = 200
|
||||
; auto: adaptive input-marker polling; N: fixed interval in microseconds (0 yields).
|
||||
MarkerPollUs = auto
|
||||
LinearInputs = true
|
||||
ElideNgxSync = true
|
||||
; Lets DLSS's last kernel write straight into the game's output buffer (DLSS 4 / K only). It takes
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
vkd3d-proton (d4r, opt-in): VKD3D_D4R_LINEAR_RGBA8=WxH also creates RGBA8 UNORM textures of exactly that size with
|
||||
unordered access linear and exportable (with VKD3D_D4R_LINEAR_TEXTURES=1), so the d4r shim can write DLSS's packed
|
||||
RGBA8 output in place (D4R_SHIM_INPLACE=1) instead of copying it. Every selected texture is logged (warn).
|
||||
Applies on top of 0001-0003.
|
||||
|
||||
diff --git a/libs/vkd3d/resource.c b/libs/vkd3d/resource.c
|
||||
--- a/libs/vkd3d/resource.c
|
||||
+++ b/libs/vkd3d/resource.c
|
||||
@@ -720,7 +720,25 @@
|
||||
desc->DepthOrArraySize != 1 || desc->SampleDesc.Count != 1 ||
|
||||
(desc->Flags & D3D12_RESOURCE_FLAG_ALLOW_DEPTH_STENCIL))
|
||||
return false;
|
||||
- return format->vk_format == VK_FORMAT_R16G16B16A16_SFLOAT || format->vk_format == VK_FORMAT_A2B10G10R10_UNORM_PACK32;
|
||||
+ if (format->vk_format == VK_FORMAT_R16G16B16A16_SFLOAT || format->vk_format == VK_FORMAT_A2B10G10R10_UNORM_PACK32)
|
||||
+ return true;
|
||||
+ /* VKD3D_D4R_LINEAR_RGBA8=WxH (opt-in, with VKD3D_D4R_LINEAR_TEXTURES=1): also RGBA8 UNORM textures of exactly
|
||||
+ * that size with unordered access, the shape of a DLSS output, so the shim can write DLSS's packed RGBA8 result
|
||||
+ * in place. Other RGBA8 targets keep optimal tiling. Every selected texture is logged. */
|
||||
+ if (format->vk_format == VK_FORMAT_R8G8B8A8_UNORM && (desc->Flags & D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS))
|
||||
+ {
|
||||
+ static unsigned int selected;
|
||||
+ unsigned int width = 0, height = 0;
|
||||
+ char size[32];
|
||||
+ if (vkd3d_get_env_var("VKD3D_D4R_LINEAR_RGBA8", size, sizeof(size)) &&
|
||||
+ sscanf(size, "%ux%u", &width, &height) == 2 && desc->Width == width && desc->Height == height)
|
||||
+ {
|
||||
+ WARN("d4r: linear exportable RGBA8 texture #%u (%ux%u, flags %#x).\n", ++selected, width, height,
|
||||
+ desc->Flags);
|
||||
+ return true;
|
||||
+ }
|
||||
+ }
|
||||
+ return false;
|
||||
}
|
||||
|
||||
static HRESULT vkd3d_get_image_create_info(struct d3d12_device *device,
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env bash
|
||||
# Compile K's LDR output using the same PTX semantics as the HDR native route. The packed-store
|
||||
# sidecar authenticates the produced code bytes; it is not a substitute for temporal validation.
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
OUT="$(realpath -m "${1:?usage: build_k_ldr_output.sh OUT_DIR [KERNEL]}")"
|
||||
NAME="${2:-hiluma_engine_output_depthinv_mvlo_ldr_max_v2_rel}"
|
||||
[[ "$NAME" =~ ^hiluma_engine_output_depth(inv|reg)_mv(hi|lo)_ldr_max_v2_rel$ ]] || exit 2
|
||||
: "${D4R_DLSS_DLL:?set D4R_DLSS_DLL to the matching NVIDIA DLSS library}"
|
||||
mkdir -p "$OUT"
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
python3 "$ROOT/kernels/tools/extract_dlss_ptx.py" "$D4R_DLSS_DLL" "$WORK/ptx" >/dev/null
|
||||
PTX="$(rg -l "\.entry[[:space:]]+$NAME[[:space:]]*\(" "$WORK/ptx"/*.ptx)"
|
||||
[[ -n "$PTX" && "$PTX" != *$'\n'* ]] || { echo 'kernel must appear in exactly one module' >&2; exit 2; }
|
||||
# LDR uses another CFG/parameter layout than HDR: do not apply HDR's tile rewrite by register name.
|
||||
python3 "$ROOT/kernels/native/ptx2hip.py" --as1 "$PTX" "$NAME" > "$WORK/output.hip"
|
||||
# D4R_LDR_OUTPUT_MAX_VGPR=N (opt-in): cap the kernel's VGPRs. The DLAA path (path A, 100 VGPRs alone) otherwise sets
|
||||
# the allocation of every block (101 VGPRs, 7 waves/SIMD); with 96 its spills stay in path A and Quality..UP blocks
|
||||
# (path B, 88 VGPRs alone) run at 8 waves/SIMD: depthinv_mvlo output 0.825 -> 0.776 ms at 1440p, DLAA 0.840 ->
|
||||
# 0.812, every frame byte-identical (private/notes/k1440-implementation.md, O1). Lower caps spill in path B.
|
||||
if [[ -n "${D4R_LDR_OUTPUT_MAX_VGPR:-}" ]]; then
|
||||
[[ "$D4R_LDR_OUTPUT_MAX_VGPR" =~ ^[0-9]+$ ]] || { echo 'D4R_LDR_OUTPUT_MAX_VGPR must be a number' >&2; exit 2; }
|
||||
sed -i "s/__attribute__((amdgpu_flat_work_group_size(1, 256))) void $NAME(/__attribute__((amdgpu_flat_work_group_size(1, 256), amdgpu_num_vgpr($D4R_LDR_OUTPUT_MAX_VGPR))) void $NAME(/" "$WORK/output.hip"
|
||||
grep -q "amdgpu_num_vgpr($D4R_LDR_OUTPUT_MAX_VGPR)" "$WORK/output.hip" || { echo 'VGPR cap not applied' >&2; exit 2; }
|
||||
fi
|
||||
bash "$ROOT/kernels/native/build_native.sh" "$WORK/output.hip" "$OUT/$NAME.hsaco" \
|
||||
-DD4R_NATIVE_F2I -DD4R_AS1_STORES
|
||||
python3 - "$OUT" "$NAME" <<'PY'
|
||||
from pathlib import Path
|
||||
import sys
|
||||
out, name = Path(sys.argv[1]), sys.argv[2]
|
||||
value = 14695981039346656037
|
||||
for byte in (out / (name + '.hsaco')).read_bytes():
|
||||
value = ((value ^ byte) * 1099511628211) & ((1 << 64) - 1)
|
||||
path = out / 'd4r-rgba8.txt'
|
||||
rows = path.read_text().splitlines() if path.exists() else ['# RGBA8 store capability: NAME FNV1a64 of code object']
|
||||
rows = [row for row in rows if not row.startswith(name + ' ')]
|
||||
path.write_text('\n'.join(rows + [f'{name} {value:016x}']) + '\n')
|
||||
PY
|
||||
python3 "$ROOT/kernels/tools/kernel_manifest.py" "$OUT" "$D4R_DLSS_DLL"
|
||||
@@ -4,6 +4,7 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
OUT="${1:-$ROOT/build/vulkan-staging}"
|
||||
mkdir -p "$OUT"
|
||||
glslangValidator -V --target-env vulkan1.0 -o "$OUT/read.spv" "$ROOT/kernels/support/vulkan_read.comp" >/dev/null
|
||||
glslangValidator -V --target-env vulkan1.0 -o "$OUT/wait.spv" "$ROOT/kernels/support/vulkan_wait.comp" >/dev/null
|
||||
for format in rgba16f rgba32f r11f_g11f_b10f rgb10_a2 rgba8; do
|
||||
glslangValidator -V --target-env vulkan1.0 -DOUTPUT_FORMAT="$format" \
|
||||
-o "$OUT/write_$format.spv" "$ROOT/kernels/support/vulkan_write.comp" >/dev/null
|
||||
|
||||
@@ -161,7 +161,7 @@ def main(argv):
|
||||
value = flag("Interop", key)
|
||||
if value is not None:
|
||||
env[var] = "1" if value else "0"
|
||||
value = get("Interop", "MarkerPollUs") or "200"
|
||||
value = get("Interop", "MarkerPollUs")
|
||||
dilation = get("Interop", "MotionVectorDilation")
|
||||
if dilation is not None and dilation not in ("0", "1", "2"):
|
||||
fail("[Interop] MotionVectorDilation must be 0, 1 or 2")
|
||||
|
||||
@@ -30,8 +30,10 @@ struct Feature {
|
||||
int splitSemaphore = 1;
|
||||
bool engineNetTimeline = false; // the Vulkan timeline of this test, not the native network's
|
||||
int engineNet = 0;
|
||||
void* ngxTimeline = nullptr;
|
||||
};
|
||||
static struct { int (*signalTimeline)(int, uint64_t) = nullptr; } g_engineNet;
|
||||
static struct { struct { int (*timelineSignal)(void*, uint64_t) = nullptr; } cu; } g;
|
||||
static uint64_t gpuTimeline = 0;
|
||||
static bool failSignal = false;
|
||||
static int signal(int, const VkSemaphoreSignalInfo* info) {
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
"""CPU-only input-wait cancellation, polling and profiling contracts. No HIP or Wine runs."""
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def function(source, signature):
|
||||
start = source.index(signature)
|
||||
opening = source.index('{', start)
|
||||
depth = 1
|
||||
end = opening + 1
|
||||
while depth:
|
||||
depth += (source[end] == '{') - (source[end] == '}')
|
||||
end += 1
|
||||
return source[start:end]
|
||||
|
||||
|
||||
def run(source):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
runner = Path(tmp) / 'input-wait'
|
||||
subprocess.run(['c++', '-x', 'c++', '-std=c++20', '-pthread', '-Wall', '-Wextra', '-Werror',
|
||||
'-I', str(ROOT / 'tools'), '-o', str(runner), '-'],
|
||||
input=source, text=True, check=True, capture_output=True)
|
||||
subprocess.run([str(runner)], check=True, timeout=5)
|
||||
|
||||
|
||||
class InputWaitTests(unittest.TestCase):
|
||||
def test_bridge_queues_exact_null_stream_wait_or_reports_unsupported(self):
|
||||
bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
|
||||
wait = function(bridge, 'CUresult WINAPI d4rStreamWaitValue32Eq(')
|
||||
run(r'''
|
||||
#include <cstdint>
|
||||
#include <cstring>
|
||||
#include <cassert>
|
||||
#define WINAPI
|
||||
using CUresult = int;
|
||||
using CUdeviceptr = uint64_t;
|
||||
using HIP_STREAM_WAIT_VALUE32_FN = int(*)(void*, void*, uint32_t, unsigned, uint32_t);
|
||||
constexpr int CUDA_SUCCESS = 0, CUDA_ERROR_NOT_SUPPORTED = 801;
|
||||
int context_setup_result = 0, supported = 1, waitResult = 0, queued = 0;
|
||||
bool hasWait = true;
|
||||
void ensure_context() {}
|
||||
void* d4r_ngx_stream(void* stream) { return stream; } // D4R_NGX_STREAM_PRIORITY off
|
||||
void tracef(const char*, ...) {}
|
||||
bool trace_verbose() { return false; }
|
||||
int device(int* out) { *out = 3; return 0; }
|
||||
int attribute(int* out, int attr, int dev) { assert(attr == 10013 && dev == 3); *out = supported; return 0; }
|
||||
int queue(void* stream, void* ptr, uint32_t value, unsigned flags, uint32_t mask) {
|
||||
assert(stream == nullptr && ptr == reinterpret_cast<void*>(0x10008));
|
||||
assert(value == 0xffffffffu && flags == 1 && mask == 0xffffffffu);
|
||||
if (waitResult == 0) ++queued;
|
||||
return waitResult;
|
||||
}
|
||||
void* hip_symbol(const char* name) {
|
||||
if (strcmp(name, "hipStreamWaitValue32") == 0) return hasWait ? reinterpret_cast<void*>(queue) : nullptr;
|
||||
if (strcmp(name, "hipDeviceGetAttribute") == 0) return reinterpret_cast<void*>(attribute);
|
||||
if (strcmp(name, "hipGetDevice") == 0) return reinterpret_cast<void*>(device);
|
||||
return nullptr;
|
||||
}
|
||||
''' + wait + r'''
|
||||
int main() {
|
||||
assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 0 && queued == 1);
|
||||
supported = 0;
|
||||
assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 801 && queued == 1);
|
||||
supported = 1; hasWait = false;
|
||||
assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 801 && queued == 1);
|
||||
hasWait = true; waitResult = 1;
|
||||
assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 801 && queued == 1);
|
||||
context_setup_result = 700;
|
||||
assert(d4rStreamWaitValue32Eq(0x10008, 0xffffffffu) == 700 && queued == 1);
|
||||
}
|
||||
''')
|
||||
|
||||
def test_readiness_discard_retirement_timeout_wrap_and_adaptive_pacing(self):
|
||||
run(r'''
|
||||
#include "d4r_input_wait.h"
|
||||
#include <cassert>
|
||||
int main() {
|
||||
using A = D4rInputWaitAbort;
|
||||
assert(d4r_input_wait_abort(7, 6, 6, false, 100) == A::None);
|
||||
assert(d4r_input_wait_abort(7, 6, 8, false, 100) == A::Discarded);
|
||||
assert(d4r_input_wait_abort(7, 7, 8, false, 100) == A::None); // slot recheck wins
|
||||
assert(d4r_input_wait_abort(7, 7, 7, true, 100) == A::Retired);
|
||||
assert(d4r_input_wait_abort(7, 6, 6, false, 5000001) == A::Timeout);
|
||||
assert(d4r_input_wait_abort(7, 7, 7, false, 5000001) == A::Timeout); // broken wait after input
|
||||
assert(d4r_input_wait_abort(0xffffffffu, 3, 0, false, 100) == A::Discarded);
|
||||
assert(d4r_input_wait_abort(0, 0xffffffffu, 0xffffffffu, false, 100) == A::None);
|
||||
assert(d4r_input_wait_abort(0, 0xffffffffu, 1, false, 100) == A::Discarded);
|
||||
assert(d4r_input_wait_abort(0, 0, 1, false, 100) == A::None);
|
||||
assert(d4r_marker_poll_delay_us(2000, 0) == 200);
|
||||
assert(d4r_marker_poll_delay_us(2000, 1800) == 50);
|
||||
assert(d4r_marker_poll_delay_us(2000, 1850) == 0);
|
||||
assert(d4r_marker_poll_delay_us(2000, 2149) == 0);
|
||||
assert(d4r_marker_poll_delay_us(2000, 2150) == 20);
|
||||
assert(d4r_marker_poll_delay_us(2000, 3750) == 200);
|
||||
assert(d4r_marker_poll_delay_us(0, 5000) == 200); // no warm prediction
|
||||
}
|
||||
''')
|
||||
|
||||
def test_actual_release_and_drain_helpers_preserve_real_markers_and_ownership(self):
|
||||
shim = (ROOT / 'tools/d4r_nvngx_shim.cpp').read_text()
|
||||
release = function(shim, 'static void release_ngx_input_wait_locked(')
|
||||
complete = function(shim, 'static bool complete_ngx_input_wait(Feature& feature, uint32_t frame, bool abort)\n{')
|
||||
run(r'''
|
||||
#include <atomic>
|
||||
#include <cassert>
|
||||
#include <mutex>
|
||||
#include <thread>
|
||||
#include <future>
|
||||
void logf(const char*, ...) {}
|
||||
void MemoryBarrier() { std::atomic_thread_fence(std::memory_order_seq_cst); }
|
||||
struct Feature {
|
||||
std::atomic<bool> ngxGpuWait{true}, retiring{false};
|
||||
void* ngxTimeline = nullptr;
|
||||
std::mutex ngxWaitMutex;
|
||||
bool ngxWaitActive = true, ngxWaitCancelled = false;
|
||||
uint32_t ngxWaitFrame = 7;
|
||||
int ngxWaitSlot = 1;
|
||||
struct { void* mapped; } ngxInputSync;
|
||||
unsigned evaluatedFrames = 100;
|
||||
};
|
||||
uint32_t words[4] = {0, 3, 4, 5}, markers[4] = {6, 3, 4, 5};
|
||||
Feature* current;
|
||||
int drains = 0, drainResult = 0;
|
||||
int sync() {
|
||||
++drains;
|
||||
assert(words[2] == 7); // drain may only run after the CPU released the blocked queue
|
||||
assert(current->ngxWaitActive); // ticket and input ownership survive the drain
|
||||
return drainResult;
|
||||
}
|
||||
struct { struct { int(*ctxSynchronize)() = sync; void(*timelineCancel)(void*) = nullptr; } cu; } g;
|
||||
''' + release + '\n' + complete + r'''
|
||||
int main() {
|
||||
Feature f; f.ngxInputSync.mapped = words; current = &f;
|
||||
assert(complete_ngx_input_wait(f, 8, true)); // another frame cannot release this ticket
|
||||
assert(drains == 0 && words[2] == 4);
|
||||
// An independent thread can release a blocked host submission without invoking HIP.
|
||||
auto blocked = std::async(std::launch::async, [&] {
|
||||
for (;;) {
|
||||
{ std::lock_guard<std::mutex> lock(f.ngxWaitMutex); if (words[2] == 7) return; }
|
||||
std::this_thread::yield();
|
||||
}
|
||||
});
|
||||
{ std::lock_guard<std::mutex> lock(f.ngxWaitMutex);
|
||||
release_ngx_input_wait_locked(f, "discard", false); }
|
||||
blocked.get();
|
||||
assert(drains == 0 && f.ngxWaitCancelled && f.ngxGpuWait);
|
||||
assert(words[0] == 0 && words[1] == 3 && words[3] == 5);
|
||||
assert(markers[0] == 6 && markers[2] == 4); // never counterfeit game readiness
|
||||
assert(!complete_ngx_input_wait(f, 7, true));
|
||||
assert(drains == 1 && !f.ngxWaitActive && f.evaluatedFrames == 0);
|
||||
assert(complete_ngx_input_wait(f, 7, true) && drains == 1); // idempotent retirement
|
||||
f.ngxWaitActive = true; f.ngxWaitCancelled = false; f.evaluatedFrames = 2;
|
||||
assert(complete_ngx_input_wait(f, 7, false)); // output already drained: no extra sync
|
||||
assert(drains == 1 && !f.ngxWaitActive && f.evaluatedFrames == 2);
|
||||
f.ngxWaitActive = true; f.ngxWaitCancelled = false;
|
||||
{ std::lock_guard<std::mutex> lock(f.ngxWaitMutex);
|
||||
release_ngx_input_wait_locked(f, "timeout", true); }
|
||||
assert(!f.ngxGpuWait);
|
||||
drainResult = 700;
|
||||
assert(!complete_ngx_input_wait(f, 7, true));
|
||||
assert(f.retiring && !f.ngxWaitActive && drains == 2);
|
||||
}
|
||||
''')
|
||||
|
||||
def test_thread_clock_coarse_or_unavailable_reports_na(self):
|
||||
bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
|
||||
clock = function(bridge, 'double WINAPI d4rThreadCpuTimeMs(void)')
|
||||
run(r'''
|
||||
#include <time.h>
|
||||
#include <cassert>
|
||||
#define WINAPI
|
||||
long resolutionNs = 1;
|
||||
bool available = true;
|
||||
int fake_resolution(clockid_t, timespec* out) { out->tv_sec = 0; out->tv_nsec = resolutionNs; return 0; }
|
||||
int timestamp(clockid_t, timespec* out) { out->tv_sec = 2; out->tv_nsec = 123456789; return available ? 0 : -1; }
|
||||
#define clock_getres fake_resolution
|
||||
#define clock_gettime timestamp
|
||||
''' + clock + r'''
|
||||
int main() {
|
||||
assert(d4rThreadCpuTimeMs() == 2123.456789);
|
||||
resolutionNs = 10000000; assert(d4rThreadCpuTimeMs() == -1);
|
||||
resolutionNs = 1; available = false; assert(d4rThreadCpuTimeMs() == -1);
|
||||
}
|
||||
''')
|
||||
@@ -0,0 +1,204 @@
|
||||
"""CPU contracts for native completion and packed-output capability gating."""
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def function(source, signature):
|
||||
start = source.index(signature)
|
||||
opening = source.index('{', start)
|
||||
depth, end = 1, opening + 1
|
||||
while depth:
|
||||
depth += (source[end] == '{') - (source[end] == '}')
|
||||
end += 1
|
||||
return source[start:end]
|
||||
|
||||
|
||||
def run(source, cpp=False):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
exe = Path(tmp) / 'check'
|
||||
result = subprocess.run(['c++' if cpp else 'cc', '-x', 'c++' if cpp else 'c',
|
||||
'-std=c++20' if cpp else '-std=c11', '-pthread', '-Wall', '-Wextra', '-Werror',
|
||||
'-I', str(ROOT / 'tools'), '-o', str(exe), '-'],
|
||||
input=source, text=True, capture_output=True)
|
||||
if result.returncode: raise AssertionError(result.stderr)
|
||||
subprocess.run([str(exe)], check=True, timeout=5, cwd=tmp)
|
||||
|
||||
|
||||
class LatencyContracts(unittest.TestCase):
|
||||
def test_output_membership_filter_keeps_registry_results_exact(self):
|
||||
bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
|
||||
start = bridge.index('enum { OUTPUT_KERNEL_CAPACITY')
|
||||
end = bridge.index('/* 1 when the last output kernel', start)
|
||||
code = bridge[start:end]
|
||||
run(r'''
|
||||
#include <stdint.h>
|
||||
#include <stddef.h>
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
#include <pthread.h>
|
||||
#include <assert.h>
|
||||
#define CUfunction void*
|
||||
#define R_OK 4
|
||||
static const char* fake_getenv(const char* name) { (void)name; return "/objects"; }
|
||||
static int fake_access(const char* path, int mode) { (void)path; (void)mode; return 0; }
|
||||
#define getenv fake_getenv
|
||||
#define access fake_access
|
||||
typedef struct { char name[128]; int rgba8; } NativeKernel;
|
||||
NativeKernel objects[1] = {{"hiluma_engine_output_candidate", 1}};
|
||||
NativeKernel* native_kernels = objects;
|
||||
size_t native_kernel_count = 1;
|
||||
''' + code + r'''
|
||||
int main(void) {
|
||||
void* output = (void*)(uintptr_t)0x5000;
|
||||
note_function_lookup(output, "hiluma_engine_output_candidate"); note_launch(output);
|
||||
assert(last_output_native == 1 && last_output_rgba8 == 1);
|
||||
for (uintptr_t i = 1; i < 1000; ++i) {
|
||||
void* network = (void*)(i * 16);
|
||||
if (network == output) continue;
|
||||
note_launch(network);
|
||||
assert(last_output_native == 1 && last_output_rgba8 == 1); // negatives/collisions change nothing
|
||||
}
|
||||
note_function_lookup(output, "rrlite_downsample_kernel_static_ldr"); note_launch(output);
|
||||
assert(last_output_native == 1 && last_output_rgba8 == 0); // newest lookup wins on handle reuse
|
||||
}
|
||||
''')
|
||||
|
||||
def test_completed_prefix_callback_gate(self):
|
||||
run(r'''
|
||||
#include "d4r_frame_completion.h"
|
||||
#include <cassert>
|
||||
int main() {
|
||||
D4rFrameCompletion completion;
|
||||
assert(!completion.can_signal(1));
|
||||
completion.admit(1); completion.admit(2); completion.admit(3);
|
||||
assert(completion.can_signal(1) && !completion.can_signal(2));
|
||||
assert(completion.retire(3) == 0); // later discard does not cross unfinished work
|
||||
assert(!completion.can_signal(3));
|
||||
assert(completion.retire(1) == 1 && completion.can_signal(2));
|
||||
assert(completion.retire(2) == 3 && !completion.can_signal(3));
|
||||
}
|
||||
''', cpp=True)
|
||||
|
||||
def test_actual_native_callback_cancellation_monotonicity_and_detached_cleanup(self):
|
||||
bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
|
||||
start = bridge.index('typedef struct', bridge.index("/* NGX's null-stream completion"))
|
||||
end = bridge.index('} D4rNgxTimeline;', start) + len('} D4rNgxTimeline;')
|
||||
definitions = bridge[start:end]
|
||||
bodies = '\n'.join(function(bridge, sig) for sig in (
|
||||
'int WINAPI d4rNgxTimelineSignal(', 'static void d4r_ngx_completed(',
|
||||
'CUresult WINAPI d4rNgxTimelinePrepare(', 'void WINAPI d4rNgxTimelineCancel(',
|
||||
'CUresult WINAPI d4rNgxTimelineQueue(', 'void WINAPI d4rNgxTimelineDestroy(',
|
||||
'CUresult WINAPI d4rNgxTimelineWait('))
|
||||
run(r'''
|
||||
#define _POSIX_C_SOURCE 200809L
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <pthread.h>
|
||||
#include <time.h>
|
||||
#include <errno.h>
|
||||
#include <assert.h>
|
||||
#define WINAPI
|
||||
#define CUDA_SUCCESS 0
|
||||
#define CUDA_ERROR_NOT_READY 600
|
||||
#define CUDA_ERROR_NOT_SUPPORTED 801
|
||||
#define CUDA_ERROR_UNKNOWN 999
|
||||
#define CUresult int
|
||||
typedef struct { int sType; const void* pNext; uint64_t semaphore, value; } D4rVkSemaphoreSignalInfo;
|
||||
typedef int (*VK_SIGNAL_SEMAPHORE_FN)(void*, const D4rVkSemaphoreSignalInfo*);
|
||||
typedef int (*HIP_LAUNCH_HOST_FN)(void*, void(*)(void*), void*);
|
||||
static void* d4r_ngx_stream(void* stream) { return stream; } /* D4R_NGX_STREAM_PRIORITY off */
|
||||
void (*callback)(void*); void* argument;
|
||||
uint64_t signalled = 0; int signals = 0, launchResult = 0;
|
||||
int signal_fake(void* device, const D4rVkSemaphoreSignalInfo* info) {
|
||||
(void)device; assert(info->value > signalled); signalled = info->value; ++signals; return 0;
|
||||
}
|
||||
static void* complete_later(void* unused) {
|
||||
(void)unused; struct timespec pause = {0, 20000000}; nanosleep(&pause, NULL);
|
||||
callback(argument); return NULL;
|
||||
}
|
||||
int launch_fake(void* stream, void(*fn)(void*), void* arg) {
|
||||
assert(stream == NULL);
|
||||
if (!launchResult) { callback = fn; argument = arg; }
|
||||
return launchResult;
|
||||
}
|
||||
''' + definitions + '\n' + bodies + r'''
|
||||
int main(void) {
|
||||
D4rNgxTimeline* t = calloc(1, sizeof(*t));
|
||||
assert(pthread_mutex_init(&t->lock, NULL) == 0);
|
||||
pthread_condattr_t attr; assert(pthread_condattr_init(&attr) == 0);
|
||||
assert(pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) == 0);
|
||||
assert(pthread_cond_init(&t->completed, &attr) == 0); pthread_condattr_destroy(&attr);
|
||||
t->signal = signal_fake; t->launch = launch_fake;
|
||||
assert(d4rNgxTimelinePrepare(t) == 0);
|
||||
assert(d4rNgxTimelineQueue(t, 1) == 0);
|
||||
assert(d4rNgxTimelinePrepare(t) == 600 && d4rNgxTimelineQueue(t, 2) == 600);
|
||||
d4rNgxTimelineCancel(t); callback(argument); assert(signals == 0 && t->busy == 0);
|
||||
assert(d4rNgxTimelineWait(t) == 0);
|
||||
assert(d4rNgxTimelineSignal(t, 1) == 0 && signals == 1);
|
||||
assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 2) == 0);
|
||||
callback(argument); assert(signalled == 2 && signals == 2);
|
||||
assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 3) == 0);
|
||||
assert(d4rNgxTimelineSignal(t, 4) == 0); callback(argument);
|
||||
assert(signalled == 4 && signals == 3); // late callback never signals backwards
|
||||
launchResult = 1;
|
||||
assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 5) == 801 && t->busy == 0);
|
||||
launchResult = 0;
|
||||
assert(d4rNgxTimelineQueue(t, 5) == 0);
|
||||
pthread_t worker; assert(pthread_create(&worker, NULL, complete_later, NULL) == 0);
|
||||
assert(d4rNgxTimelineWait(t) == 0); pthread_join(worker, NULL);
|
||||
assert(signalled == 5 && signals == 4);
|
||||
assert(d4rNgxTimelinePrepare(t) == 0 && d4rNgxTimelineQueue(t, 6) == 0);
|
||||
d4rNgxTimelineDestroy(t); callback(argument); // ticket outlives owner, cancellation forbids a signal
|
||||
assert(signals == 4);
|
||||
}
|
||||
''')
|
||||
|
||||
def test_all_half_values_pack_like_the_reference_unorm_conversion(self):
|
||||
header = (ROOT / 'kernels/tex/tex_common.h').read_text()
|
||||
body = function(header, '__attribute__((device)) static inline uint32_t redirect_unorm(')
|
||||
body = body.replace('__attribute__((device)) ', '')
|
||||
run(r'''
|
||||
#include <cstdint>
|
||||
#include <cmath>
|
||||
#include <cassert>
|
||||
''' + body + r'''
|
||||
int main() {
|
||||
for (uint32_t bits = 0; bits < 65536; ++bits) {
|
||||
const int exponent = (bits >> 10) & 31, mantissa = bits & 1023;
|
||||
double value = exponent == 0 ? std::ldexp(double(mantissa), -24)
|
||||
: exponent == 31 ? (mantissa ? NAN : INFINITY)
|
||||
: std::ldexp(1.0 + double(mantissa) / 1024.0, exponent - 15);
|
||||
if (bits & 0x8000) value = -value;
|
||||
value = value > 0.0 ? value : 0.0;
|
||||
value = value < 1.0 ? value : 1.0;
|
||||
const uint32_t expected = uint32_t(std::floor(value * 255.0 + 0.5));
|
||||
assert(redirect_unorm(uint16_t(bits), 255.0f) == expected);
|
||||
}
|
||||
}
|
||||
''', cpp=True)
|
||||
|
||||
def test_packed_capability_requires_matching_code_object_bytes(self):
|
||||
bridge = (ROOT / 'tools/wine_nvcuda_bridge.c').read_text()
|
||||
body = function(bridge, 'static int native_rgba8_verified(')
|
||||
run(r'''
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
#include <assert.h>
|
||||
''' + body + r'''
|
||||
int main(void) {
|
||||
assert(!native_rgba8_verified("/does/not/exist", "output"));
|
||||
FILE* f = fopen("output.hsaco", "wb"); assert(f != NULL); fputs("abc", f); fclose(f);
|
||||
assert(!native_rgba8_verified(".", "output"));
|
||||
f = fopen("d4r-rgba8.txt", "w"); fputs("output e71fa2190541574b\n", f); fclose(f);
|
||||
assert(native_rgba8_verified(".", "output"));
|
||||
f = fopen("output.hsaco", "ab"); fputc('d', f); fclose(f);
|
||||
assert(!native_rgba8_verified(".", "output"));
|
||||
assert(!native_rgba8_verified(".", "other"));
|
||||
}
|
||||
''')
|
||||
@@ -35,9 +35,12 @@ class ShimConfigTests(unittest.TestCase):
|
||||
ini = Path(temporary) / 'd4r.ini'
|
||||
env = {key: value for key, value in os.environ.items() if not key.startswith('D4R_')}
|
||||
cmd = ['python3', str(ROOT / 'scripts/d4r_config.py'), '--config', str(ini)]
|
||||
for value, expected in ((None, '200'), ('auto', '200'), ('200', '200'), ('0', '0'), ('75', '75')):
|
||||
for value, expected in ((None, None), ('auto', None), ('200', '200'), ('0', '0'), ('75', '75')):
|
||||
ini.write_text('[Interop]\n' + (f'MarkerPollUs = {value}\n' if value is not None else ''))
|
||||
result = subprocess.run(cmd, env=env, text=True, capture_output=True, check=True)
|
||||
if expected is None:
|
||||
self.assertNotIn('export D4R_SHIM_MARKER_POLL_US=', result.stdout)
|
||||
else:
|
||||
self.assertIn(f'export D4R_SHIM_MARKER_POLL_US={expected}\n', result.stdout)
|
||||
# An explicit environment variable keeps its existing precedence.
|
||||
result = subprocess.run(cmd, env=dict(env, D4R_SHIM_MARKER_POLL_US='35'),
|
||||
|
||||
@@ -1125,9 +1125,9 @@ int main(int argc, char** argv)
|
||||
std::fprintf(stderr, "RGB10A2 output cannot be combined with RGBA8 or the quality scene\n");
|
||||
return 2;
|
||||
}
|
||||
if (rgba8 && (qualityScene || motionScene || jitterScene || replayDir != nullptr))
|
||||
if (rgba8 && (qualityScene || replayDir != nullptr))
|
||||
{
|
||||
std::fprintf(stderr, "RGBA8 format probe requires the static synthetic scene\n");
|
||||
std::fprintf(stderr, "RGBA8 format probe supports static, motion and jitter scenes\n");
|
||||
return 2;
|
||||
}
|
||||
const bool temporal = std::getenv("D4R_HARNESS_TEMPORAL") != nullptr || motionScene || jitterScene || qualityScene ||
|
||||
@@ -1226,6 +1226,8 @@ int main(int argc, char** argv)
|
||||
for (size_t index = 0; index < color.size(); ++index)
|
||||
color8[index] = static_cast<uint8_t>(std::lround(
|
||||
std::clamp(static_cast<float>(std::bit_cast<_Float16>(color[index])), 0.0f, 1.0f) * 255.0f));
|
||||
if (std::getenv("D4R_HARNESS_RGBA8_RAMP") != nullptr)
|
||||
for (size_t i = 0; i < color8.size(); ++i) color8[i] = static_cast<uint8_t>(i);
|
||||
outputInit8.resize(static_cast<size_t>(outWidth) * outHeight * 4);
|
||||
}
|
||||
|
||||
@@ -1421,6 +1423,17 @@ int main(int argc, char** argv)
|
||||
: rgb10 ? DXGI_FORMAT_R10G10B10A2_UNORM
|
||||
: DXGI_FORMAT_R16G16B16A16_FLOAT,
|
||||
D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS);
|
||||
// Moving/jittered RGBA8 fixtures quantize the same canonical half input as the initial upload.
|
||||
// Never upload the half byte layout into an RGBA8 resource: that would make a format test silently
|
||||
// exercise different content instead of checking the conversion/packing route used by the game.
|
||||
const auto update_color = [&] {
|
||||
if (rgba8)
|
||||
for (size_t i = 0; i < color.size(); ++i)
|
||||
color8[i] = static_cast<uint8_t>(std::lround(std::clamp(
|
||||
static_cast<float>(std::bit_cast<_Float16>(color[i])), 0.0f, 1.0f) * 255.0f));
|
||||
update_texture(colorTexture, rgba8 ? static_cast<const void*>(color8.data())
|
||||
: static_cast<const void*>(color.data()), resourceWidth * (rgba8 ? 4 : 8), srv);
|
||||
};
|
||||
upload(colorTexture, rgba8 ? static_cast<const void*>(color8.data()) : static_cast<const void*>(color.data()),
|
||||
resourceWidth * (rgba8 ? 4 : 8), srv);
|
||||
upload(depthTexture, depth.data(), resourceWidth * 4, srv);
|
||||
@@ -1757,7 +1770,7 @@ int main(int argc, char** argv)
|
||||
color[pixel * 4 + 0] = color[pixel * 4 + 1] = color[pixel * 4 + 2] = half;
|
||||
color[pixel * 4 + 3] = 0x3c00u;
|
||||
}
|
||||
update_texture(colorTexture, color.data(), resourceWidth * 8, srv);
|
||||
update_color();
|
||||
if (frame == 1 && (panX != 0 || panY != 0))
|
||||
{
|
||||
// Current position -> previous position. Without MVLowRes DLSS
|
||||
@@ -1800,7 +1813,7 @@ int main(int argc, char** argv)
|
||||
{0.0f, 0.25f, -0.25f, -0.125f, 0.375f, 0.125f, -0.375f, 0.0f};
|
||||
const size_t phase = static_cast<size_t>(frame - 1) % jitterX.size();
|
||||
render_jittered_background(color, inWidth, inHeight, resourceWidth, jitterX[phase], jitterY[phase]);
|
||||
update_texture(colorTexture, color.data(), resourceWidth * 8, srv);
|
||||
update_color();
|
||||
d4r_ngx_set_float(parameters, "Jitter.Offset.X", jitterX[phase]);
|
||||
d4r_ngx_set_float(parameters, "Jitter.Offset.Y", jitterY[phase]);
|
||||
}
|
||||
@@ -1823,7 +1836,7 @@ int main(int argc, char** argv)
|
||||
motion[motionPixel * 2 + 0] = 0xc400u; // -4 render pixels: current position -> previous
|
||||
motion[motionPixel * 2 + 1] = 0;
|
||||
}
|
||||
update_texture(colorTexture, color.data(), resourceWidth * 8, srv);
|
||||
update_color();
|
||||
update_texture(depthTexture, depth.data(), resourceWidth * 4, srv);
|
||||
update_texture(motionTexture, motion_rows(motion.data()), motionRowBytes, srv);
|
||||
}
|
||||
@@ -2054,6 +2067,20 @@ int main(int argc, char** argv)
|
||||
LARGE_INTEGER counterStart, counterRecorded, counterEnd, counterFrequency;
|
||||
QueryPerformanceFrequency(&counterFrequency);
|
||||
QueryPerformanceCounter(&counterStart);
|
||||
// D4R_HARNESS_THRASH_MB=N: copy N MB between two private buffers ahead of the evaluation, as a game's
|
||||
// own rendering does, so DLSS starts with its weights and history evicted from L2 and Infinity Cache
|
||||
// instead of finding them warm from the previous back-to-back frame.
|
||||
static const UINT64 thrashBytes = std::getenv("D4R_HARNESS_THRASH_MB") != nullptr
|
||||
? std::clamp<UINT64>(std::strtoull(std::getenv("D4R_HARNESS_THRASH_MB"), nullptr, 10), 0, 1024) << 20 : 0;
|
||||
static ID3D12Resource* thrashSource = nullptr;
|
||||
static ID3D12Resource* thrashTarget = nullptr;
|
||||
if (thrashBytes != 0 && thrashSource == nullptr)
|
||||
{
|
||||
thrashSource = create_buffer(D3D12_HEAP_TYPE_DEFAULT, thrashBytes, D3D12_RESOURCE_STATE_COMMON);
|
||||
thrashTarget = create_buffer(D3D12_HEAP_TYPE_DEFAULT, thrashBytes, D3D12_RESOURCE_STATE_COMMON);
|
||||
}
|
||||
if (thrashSource != nullptr && thrashTarget != nullptr)
|
||||
g_list->CopyBufferRegion(thrashTarget, 0, thrashSource, 0, thrashBytes);
|
||||
result = evaluate(g_list, feature, parameters, nullptr);
|
||||
QueryPerformanceCounter(&counterRecorded);
|
||||
if (const char* delay = std::getenv("D4R_HARNESS_DELAY_SUBMIT_MS"))
|
||||
|
||||
@@ -11,6 +11,11 @@ class D4rFrameCompletion
|
||||
public:
|
||||
void admit(uint32_t frame) { pending_.push_back({frame, false}); }
|
||||
|
||||
// A native completion callback can release this value without crossing an unfinished producer
|
||||
// only when it is the first admitted frame still outstanding. Retirement itself stays on the
|
||||
// worker, which also folds any later discarded frames into the completed prefix.
|
||||
bool can_signal(uint32_t frame) const { return !pending_.empty() && pending_.front().frame == frame; }
|
||||
|
||||
uint64_t retire(uint32_t frame)
|
||||
{
|
||||
for (auto& pending : pending_)
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
#pragma once
|
||||
#include <cstdint>
|
||||
|
||||
// The original marker remains the authority for whether a command list ran. A CPU release of the
|
||||
// separate, coherent GPU wait word must never make discarded input look ready to the worker.
|
||||
enum class D4rInputWaitAbort { None, Retired, Discarded, Timeout };
|
||||
|
||||
inline D4rInputWaitAbort d4r_input_wait_abort(uint32_t frame, uint32_t slotMarker, uint32_t aggregate,
|
||||
bool retiring, double elapsedUs)
|
||||
{
|
||||
if (retiring)
|
||||
return D4rInputWaitAbort::Retired;
|
||||
if (slotMarker != frame && static_cast<int32_t>(aggregate - frame) > 0)
|
||||
return D4rInputWaitAbort::Discarded;
|
||||
// Also covers a stream wait that failed to observe an input word already written by Vulkan.
|
||||
if (elapsedUs > 5000000.0)
|
||||
return D4rInputWaitAbort::Timeout;
|
||||
return D4rInputWaitAbort::None;
|
||||
}
|
||||
|
||||
inline unsigned d4r_marker_poll_delay_us(double predictedUs, double elapsedUs)
|
||||
{
|
||||
const double remaining = predictedUs - 150.0 - elapsedUs;
|
||||
if (remaining > 20.0)
|
||||
return static_cast<unsigned>(remaining < 200.0 ? remaining : 200.0);
|
||||
if (elapsedUs < predictedUs + 150.0)
|
||||
return 0; // yield during the final 150 us and a short late window
|
||||
const double late = 20.0 * (1.0 + (elapsedUs - predictedUs - 150.0) / 160.0);
|
||||
return static_cast<unsigned>(late < 200.0 ? late : 200.0);
|
||||
}
|
||||
+752
-53
File diff suppressed because it is too large.
Load diff
@@ -16,6 +16,10 @@ struct D4rVkStaging {
|
||||
VkDescriptorSetLayout readSet = VK_NULL_HANDLE, writeSet = VK_NULL_HANDLE;
|
||||
VkPipelineLayout readLayout = VK_NULL_HANDLE, writeLayout = VK_NULL_HANDLE;
|
||||
VkPipeline readPipeline = VK_NULL_HANDLE;
|
||||
// D4R_SHIM_GPU_COMPLETION: one storage buffer of words, push constants {index, value, limit}
|
||||
VkDescriptorSetLayout waitSet = VK_NULL_HANDLE;
|
||||
VkPipelineLayout waitLayout = VK_NULL_HANDLE;
|
||||
VkPipeline waitPipeline = VK_NULL_HANDLE;
|
||||
struct OutputPipeline { VkFormat format; VkPipeline pipeline; } outputs[5] = {};
|
||||
PFN_vkCreateSampler createSampler = nullptr;
|
||||
PFN_vkDestroySampler destroySampler = nullptr;
|
||||
@@ -81,7 +85,7 @@ struct D4rVkStaging {
|
||||
set.bindingCount = 2; set.pBindings = bindings;
|
||||
auto& setLayout = output ? writeSet : readSet;
|
||||
if (createSetLayout(d, &set, nullptr, &setLayout) != VK_SUCCESS) return false;
|
||||
VkPushConstantRange range = {VK_SHADER_STAGE_COMPUTE_BIT, 0, output ? 8u : 12u};
|
||||
VkPushConstantRange range = {VK_SHADER_STAGE_COMPUTE_BIT, 0, output ? 8u : 16u};
|
||||
VkPipelineLayoutCreateInfo layout = {VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO};
|
||||
layout.setLayoutCount = 1; layout.pSetLayouts = &setLayout;
|
||||
layout.pushConstantRangeCount = 1; layout.pPushConstantRanges = ⦥
|
||||
@@ -89,6 +93,44 @@ struct D4rVkStaging {
|
||||
}
|
||||
return pipeline(d4r_vk_read, sizeof(d4r_vk_read), readLayout, readPipeline);
|
||||
}
|
||||
// Created on first use; false leaves GPU completion off.
|
||||
bool wait_ready() {
|
||||
if (waitPipeline) return true;
|
||||
VkDescriptorSetLayoutBinding binding = {0, VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1, VK_SHADER_STAGE_COMPUTE_BIT, nullptr};
|
||||
VkDescriptorSetLayoutCreateInfo set = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO};
|
||||
set.bindingCount = 1; set.pBindings = &binding;
|
||||
if (!waitSet && createSetLayout(device, &set, nullptr, &waitSet) != VK_SUCCESS) return false;
|
||||
VkPushConstantRange range = {VK_SHADER_STAGE_COMPUTE_BIT, 0, 12};
|
||||
VkPipelineLayoutCreateInfo layout = {VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO};
|
||||
layout.setLayoutCount = 1; layout.pSetLayouts = &waitSet;
|
||||
layout.pushConstantRangeCount = 1; layout.pPushConstantRanges = ⦥
|
||||
if (!waitLayout && createLayout(device, &layout, nullptr, &waitLayout) != VK_SUCCESS) return false;
|
||||
return pipeline(d4r_vk_wait, sizeof(d4r_vk_wait), waitLayout, waitPipeline);
|
||||
}
|
||||
bool wait_set(D4rVkStagingSet& result, VkBuffer buffer, VkDeviceSize bytes) {
|
||||
if (!wait_ready()) return false;
|
||||
VkDescriptorPoolSize size = {VK_DESCRIPTOR_TYPE_STORAGE_BUFFER, 1};
|
||||
VkDescriptorPoolCreateInfo pool = {VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO};
|
||||
pool.maxSets = 1; pool.poolSizeCount = 1; pool.pPoolSizes = &size;
|
||||
if (!result.pool && createPool(device, &pool, nullptr, &result.pool) != VK_SUCCESS) return false;
|
||||
VkDescriptorSetAllocateInfo alloc = {VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO};
|
||||
alloc.descriptorPool = result.pool; alloc.descriptorSetCount = 1; alloc.pSetLayouts = &waitSet;
|
||||
if (!result.set && allocateSets(device, &alloc, &result.set) != VK_SUCCESS) return false;
|
||||
VkDescriptorBufferInfo storage = {buffer, 0, bytes};
|
||||
VkWriteDescriptorSet write = {VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET};
|
||||
write.dstSet = result.set; write.dstBinding = 0; write.descriptorCount = 1;
|
||||
write.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER; write.pBufferInfo = &storage;
|
||||
updateSets(device, 1, &write, 0, nullptr);
|
||||
result.pipeline = waitPipeline;
|
||||
return true;
|
||||
}
|
||||
void record_wait(VkCommandBuffer cmd, const D4rVkStagingSet& set, uint32_t index, uint32_t value, uint32_t limit) {
|
||||
bindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, waitPipeline);
|
||||
bindSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, waitLayout, 0, 1, &set.set, 0, nullptr);
|
||||
const uint32_t constants[] = {index, value, limit};
|
||||
push(cmd, waitLayout, VK_SHADER_STAGE_COMPUTE_BIT, 0, 12, constants);
|
||||
dispatch(cmd, 1, 1, 1);
|
||||
}
|
||||
VkPipeline output_pipeline(VkFormat format) {
|
||||
for (auto& item : outputs) if (item.pipeline && item.format == format) return item.pipeline;
|
||||
const uint32_t* code = nullptr; size_t bytes = 0;
|
||||
@@ -135,12 +177,12 @@ struct D4rVkStaging {
|
||||
updateSets(device, 2, writes, 0, nullptr);
|
||||
return true;
|
||||
}
|
||||
void record(VkCommandBuffer cmd, const D4rVkStagingSet& set, uint32_t width, uint32_t height, uint32_t inputMode = 0) {
|
||||
void record(VkCommandBuffer cmd, const D4rVkStagingSet& set, uint32_t width, uint32_t height, uint32_t inputMode = 0, uint32_t stride = 0) {
|
||||
const auto layout = set.output ? writeLayout : readLayout;
|
||||
bindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, set.pipeline);
|
||||
bindSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, layout, 0, 1, &set.set, 0, nullptr);
|
||||
const uint32_t dimensions[] = {width, height, inputMode};
|
||||
push(cmd, layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, set.output ? 8 : 12, dimensions);
|
||||
const uint32_t dimensions[] = {width, height, inputMode, stride ? stride : width};
|
||||
push(cmd, layout, VK_SHADER_STAGE_COMPUTE_BIT, 0, set.output ? 8 : 16, dimensions);
|
||||
dispatch(cmd, (width+7)/8, (height+7)/8, 1);
|
||||
}
|
||||
void free(D4rVkStagingSet& set) {
|
||||
@@ -149,6 +191,9 @@ struct D4rVkStaging {
|
||||
}
|
||||
void destroy() {
|
||||
if (readPipeline) destroyPipeline(device, readPipeline, nullptr);
|
||||
if (waitPipeline) destroyPipeline(device, waitPipeline, nullptr);
|
||||
if (waitLayout) destroyLayout(device, waitLayout, nullptr);
|
||||
if (waitSet) destroySetLayout(device, waitSet, nullptr);
|
||||
for (auto& item : outputs) if (item.pipeline) destroyPipeline(device, item.pipeline, nullptr);
|
||||
if (readLayout) destroyLayout(device, readLayout, nullptr);
|
||||
if (writeLayout) destroyLayout(device, writeLayout, nullptr);
|
||||
|
||||
+384
-24
@@ -11,6 +11,7 @@
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <unistd.h>
|
||||
#include <time.h>
|
||||
#include "d4r_native_selection.h"
|
||||
#include "d4r_motion_dilation.h"
|
||||
#ifdef D4R_MOTION_KERNELS
|
||||
@@ -366,11 +367,41 @@ typedef struct
|
||||
char name[128];
|
||||
uint64_t hash;
|
||||
int linked;
|
||||
int rgba8;
|
||||
} NativeKernel;
|
||||
|
||||
static NativeKernel* native_kernels;
|
||||
static size_t native_kernel_count;
|
||||
static char native_source[1024];
|
||||
|
||||
/* Packed stores are a code-object capability, not a property of the kernel's name. A legacy native
|
||||
kernel ignores the R4 redirect tag. The sidecar must therefore name the exact bytes installed
|
||||
alongside the PTX compatibility manifest; the per-process cache retains those same bytes. */
|
||||
static int native_rgba8_verified(const char* directory, const char* name)
|
||||
{
|
||||
char path[1400], row[256], entry[128];
|
||||
unsigned long long expected = 0;
|
||||
snprintf(path, sizeof(path), "%s/d4r-rgba8.txt", directory);
|
||||
FILE* manifest = fopen(path, "r");
|
||||
if (manifest == NULL) return 0;
|
||||
int found = 0;
|
||||
while (fgets(row, sizeof(row), manifest) != NULL)
|
||||
if (sscanf(row, "%127s %llx", entry, &expected) == 2 && strcmp(entry, name) == 0)
|
||||
{ found = 1; break; }
|
||||
fclose(manifest);
|
||||
if (!found) return 0;
|
||||
snprintf(path, sizeof(path), "%s/%s.hsaco", directory, name);
|
||||
FILE* object = fopen(path, "rb");
|
||||
if (object == NULL) return 0;
|
||||
uint64_t hash = UINT64_C(14695981039346656037);
|
||||
unsigned char bytes[4096];
|
||||
size_t count;
|
||||
while ((count = fread(bytes, 1, sizeof(bytes), object)) != 0)
|
||||
for (size_t i = 0; i < count; ++i) hash = (hash ^ bytes[i]) * UINT64_C(1099511628211);
|
||||
const int valid = !ferror(object) && hash == (uint64_t)expected;
|
||||
fclose(object);
|
||||
return valid;
|
||||
}
|
||||
static char native_served[1024];
|
||||
static pthread_mutex_t native_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||
|
||||
@@ -506,6 +537,7 @@ static void prepare_native_kernels(const char* cache_home)
|
||||
snprintf(kernel->name, sizeof(kernel->name), "%s", name);
|
||||
kernel->hash = hash;
|
||||
kernel->linked = 0;
|
||||
kernel->rgba8 = native_rgba8_verified(source, name);
|
||||
}
|
||||
fclose(manifest);
|
||||
if (!d4r_native_swin_encoders_enabled())
|
||||
@@ -733,9 +765,22 @@ static struct
|
||||
{
|
||||
CUfunction function;
|
||||
int native;
|
||||
int rgba8;
|
||||
} output_kernels[OUTPUT_KERNEL_CAPACITY]; /* the latest lookups, newest last */
|
||||
static pthread_mutex_t output_kernel_lock = PTHREAD_MUTEX_INITIALIZER;
|
||||
static int last_output_native = -1;
|
||||
static int last_output_rgba8;
|
||||
/* Most NGX launches are network/input kernels. A monotonic membership filter lets those bypass the
|
||||
output registry's lock and scan. Collisions only take the old lookup path; a bit is never cleared,
|
||||
so function-handle reuse, feature recreation and concurrent lookups cannot hide an output kernel. */
|
||||
static uint64_t output_kernel_bits[16];
|
||||
|
||||
static unsigned output_kernel_bit(CUfunction function)
|
||||
{
|
||||
uint64_t value = (uint64_t)(uintptr_t)function >> 4;
|
||||
value ^= value >> 17;
|
||||
return (unsigned)((value * UINT64_C(0x9e3779b97f4a7c15)) >> 54);
|
||||
}
|
||||
|
||||
static void note_function_lookup(CUfunction function, const char* name)
|
||||
{
|
||||
@@ -755,20 +800,34 @@ static void note_function_lookup(CUfunction function, const char* name)
|
||||
memmove(output_kernels, output_kernels + 1, sizeof(output_kernels) - sizeof(output_kernels[0]));
|
||||
output_kernels[OUTPUT_KERNEL_CAPACITY - 1].function = function;
|
||||
output_kernels[OUTPUT_KERNEL_CAPACITY - 1].native = native;
|
||||
int rgba8 = 0;
|
||||
for (size_t i = 0; i < native_kernel_count; ++i)
|
||||
if (strcmp(native_kernels[i].name, name) == 0) { rgba8 = native_kernels[i].rgba8; break; }
|
||||
output_kernels[OUTPUT_KERNEL_CAPACITY - 1].rgba8 = native && rgba8;
|
||||
pthread_mutex_unlock(&output_kernel_lock);
|
||||
const unsigned bit = output_kernel_bit(function);
|
||||
__atomic_fetch_or(&output_kernel_bits[bit >> 6], UINT64_C(1) << (bit & 63), __ATOMIC_RELEASE);
|
||||
}
|
||||
|
||||
static void note_launch(CUfunction function)
|
||||
/* Returns 1 + native + 2 * rgba8 when `function` is an output kernel, 0 otherwise. */
|
||||
static int note_launch(CUfunction function)
|
||||
{
|
||||
const unsigned bit = output_kernel_bit(function);
|
||||
if ((__atomic_load_n(&output_kernel_bits[bit >> 6], __ATOMIC_ACQUIRE) & (UINT64_C(1) << (bit & 63))) == 0)
|
||||
return 0;
|
||||
int kind = 0;
|
||||
pthread_mutex_lock(&output_kernel_lock);
|
||||
/* newest first, so a recycled handle resolves with its latest lookup */
|
||||
for (int i = OUTPUT_KERNEL_CAPACITY - 1; i >= 0 && output_kernels[i].function != NULL; --i)
|
||||
if (output_kernels[i].function == function)
|
||||
{
|
||||
__atomic_store_n(&last_output_native, output_kernels[i].native, __ATOMIC_RELAXED);
|
||||
__atomic_store_n(&last_output_rgba8, output_kernels[i].rgba8, __ATOMIC_RELAXED);
|
||||
kind = 1 + output_kernels[i].native + 2 * output_kernels[i].rgba8;
|
||||
break;
|
||||
}
|
||||
pthread_mutex_unlock(&output_kernel_lock);
|
||||
return kind;
|
||||
}
|
||||
|
||||
/* 1 when the last output kernel launched was d4r's native one, 0 when it was ZLUDA's compile of
|
||||
@@ -778,6 +837,11 @@ int WINAPI d4rOutputKernelNative(void)
|
||||
return __atomic_load_n(&last_output_native, __ATOMIC_RELAXED);
|
||||
}
|
||||
|
||||
int WINAPI d4rOutputKernelRgba8(void)
|
||||
{
|
||||
return __atomic_load_n(&last_output_rgba8, __ATOMIC_RELAXED);
|
||||
}
|
||||
|
||||
static void load_zluda(void)
|
||||
{
|
||||
/* Explicit experimental backend. Never silently mix module/context/array
|
||||
@@ -964,7 +1028,7 @@ CUresult WINAPI d4rDilateMotionVectors(const D4rMotionDilationParams* parameters
|
||||
D4rMotionDilationParams copy = *parameters;
|
||||
void* arguments[] = {©};
|
||||
const uint64_t pixels = (uint64_t)copy.motion_width * copy.motion_height;
|
||||
return launch(function, (unsigned)((pixels + 127) / 128), 1, 1, 128, 1, 1, 0, NULL, arguments, NULL);
|
||||
return launch(function, (unsigned)((pixels + 127) / 128), 1, 1, 128, 1, 1, 0, d4r_ngx_stream(NULL), arguments, NULL);
|
||||
#else
|
||||
return CUDA_ERROR_NOT_SUPPORTED;
|
||||
#endif
|
||||
@@ -1127,6 +1191,16 @@ static void ensure_context(void)
|
||||
pthread_once(&context_once, create_default_zluda_context);
|
||||
}
|
||||
|
||||
/* D4R_NGX_STREAM_PRIORITY=high|normal (default off): the null-stream work of NGX, the shim and this bridge
|
||||
(kernels, async copies and memsets, events, stream waits, input waits, completion callbacks) moves from
|
||||
HIP's null stream to one blocking HIP stream. "high" creates it at the device's greatest priority, so its
|
||||
HSA queue (and the hardware queue it maps to) dispatches ahead of normal-priority queues; "normal" creates
|
||||
it at the default priority, the control that isolates the stream change itself. Every caller passes NULL
|
||||
and gets the same stream, so the order of the work is unchanged; being blocking, the stream also stays
|
||||
ordered with ZLUDA's own synchronous null-stream operations. Off: NULL stays HIP's null stream. */
|
||||
static CUstream d4r_ngx_stream(CUstream stream);
|
||||
static void* hip_symbol(const char* name);
|
||||
|
||||
/* Opt-in stage instrumentation. With D4R_CUDA_LAUNCH_STATS=1 every
|
||||
cuLaunchKernel is followed by a context synchronize and a summary of each
|
||||
device allocation, surface object and texture object named in the packed
|
||||
@@ -1402,15 +1476,17 @@ static CUresult write_redirect_tail(CUsurfObject object, CUdeviceptr pointer, ui
|
||||
static unsigned int next;
|
||||
unsigned char* tail = ring[next++ % 256];
|
||||
// dword 21: 0x5232 ('R2', RGBA16F texels) << 16 | pitch / 8 (0 = off); dwords 22-23: pointer.
|
||||
// Row pitches are multiples of 8, so bit 0 of `pitch` selects 'R3': R10G10B10A2_UNORM texels.
|
||||
const uint32_t word = pitch != 0 ? ((pitch & 1u) != 0 ? 0x52330000u : 0x52320000u) | ((pitch >> 3) & 0xffffu) : 0u;
|
||||
// Row pitches are multiples of 8: bit 0 selects R3 (RGB10A2), bit 1 selects R4 (RGBA8).
|
||||
const uint32_t word = pitch != 0 ? ((pitch & 2u) != 0 ? 0x52340000u :
|
||||
(pitch & 1u) != 0 ? 0x52330000u : 0x52320000u) |
|
||||
((pitch >> 3) & 0xffffu) : 0u;
|
||||
memset(tail, 0, 16);
|
||||
memcpy(tail, &word, 4);
|
||||
memcpy(tail + 4, &pointer, 8);
|
||||
CUMEMCPYHTODASYNC_FN function = (CUMEMCPYHTODASYNC_FN)find_zluda_symbol("cuMemcpyHtoDAsync_v2");
|
||||
if (function == NULL)
|
||||
return CUDA_ERROR_NOT_SUPPORTED;
|
||||
CUresult result = function((CUdeviceptr)object + 84, tail, 12, NULL);
|
||||
CUresult result = function((CUdeviceptr)object + 84, tail, 12, d4r_ngx_stream(NULL));
|
||||
if (result == CUDA_SUCCESS && !async)
|
||||
{
|
||||
CUCTX_SYNCHRONIZE_FN sync = (CUCTX_SYNCHRONIZE_FN)find_zluda_symbol("cuCtxSynchronize");
|
||||
@@ -2431,6 +2507,33 @@ CUresult WINAPI cuModuleGetFunction(CUfunction* function_out, CUmodule module, c
|
||||
return result;
|
||||
}
|
||||
|
||||
/* D4R_SHIM_GPU_COMPLETION, early form: the shim arms a completion word before NGX's evaluate. When the output
|
||||
kernel that follows is d4r's native one (which writes the game-visible buffer directly; RGBA8-capable when
|
||||
the frame needs the packed store), the word is queued right behind that kernel, ahead of NGX's trailing
|
||||
bookkeeping kernels, which the game never reads. Any other output kernel leaves it to the shim's own write
|
||||
after the output copy. Only the CUDA worker evaluates, so the arm state needs no lock. */
|
||||
static CUdeviceptr ngx_completion_pointer;
|
||||
static uint32_t ngx_completion_value;
|
||||
static int ngx_completion_rgba8, ngx_completion_fired;
|
||||
|
||||
CUresult WINAPI d4rNgxArmCompletion(CUdeviceptr pointer, uint32_t value, int rgba8)
|
||||
{
|
||||
ngx_completion_pointer = pointer;
|
||||
ngx_completion_value = value;
|
||||
ngx_completion_rgba8 = rgba8;
|
||||
ngx_completion_fired = 0;
|
||||
return CUDA_SUCCESS;
|
||||
}
|
||||
|
||||
/* 1 when the armed word was queued behind the native output kernel; disarms. */
|
||||
int WINAPI d4rNgxCompletionFired(void)
|
||||
{
|
||||
const int fired = ngx_completion_fired;
|
||||
ngx_completion_pointer = 0;
|
||||
ngx_completion_fired = 0;
|
||||
return fired;
|
||||
}
|
||||
|
||||
CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x, unsigned int grid_y,
|
||||
unsigned int grid_z, unsigned int block_x, unsigned int block_y,
|
||||
unsigned int block_z, unsigned int shared_bytes, CUstream stream,
|
||||
@@ -2439,13 +2542,11 @@ CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x,
|
||||
static CULAUNCHKERNEL_FN function;
|
||||
if (function == NULL)
|
||||
function = (CULAUNCHKERNEL_FN)find_zluda_symbol("cuLaunchKernel");
|
||||
unsigned int sequence = 0;
|
||||
pthread_mutex_lock(&instrumentation_lock);
|
||||
sequence = ++launch_sequence;
|
||||
pthread_mutex_unlock(&instrumentation_lock);
|
||||
stream = d4r_ngx_stream(stream);
|
||||
const unsigned int sequence = __atomic_add_fetch(&launch_sequence, 1, __ATOMIC_RELAXED);
|
||||
replay_dump_launch(sequence, function_handle, grid_x, grid_y, grid_z, block_x, block_y, block_z,
|
||||
shared_bytes, extra);
|
||||
note_launch(function_handle);
|
||||
const int output_kind = note_launch(function_handle);
|
||||
PendingKernelProfile* profile = function != NULL
|
||||
? begin_kernel_profile(sequence, function_handle, stream) : NULL;
|
||||
CUfunction native_function = microcuda_k_lookup(function_handle);
|
||||
@@ -2468,6 +2569,17 @@ CUresult WINAPI cuLaunchKernel(CUfunction function_handle, unsigned int grid_x,
|
||||
tracef("microcuda-k prep stats runs=%llu skips=%llu", (unsigned long long)prep, (unsigned long long)skips);
|
||||
}
|
||||
end_kernel_profile(profile, result, stream);
|
||||
if (result == CUDA_SUCCESS && ngx_completion_pointer != 0 && output_kind != 0)
|
||||
{
|
||||
const int native = ((output_kind - 1) & 1) != 0, rgba8 = (output_kind - 1) >= 2;
|
||||
typedef int (*WRITE_FN)(void*, void*, uint32_t, unsigned int);
|
||||
static WRITE_FN write;
|
||||
if (write == NULL)
|
||||
write = (WRITE_FN)hip_symbol("hipStreamWriteValue32");
|
||||
if (native && (!ngx_completion_rgba8 || rgba8) && write != NULL)
|
||||
ngx_completion_fired = write(stream, (void*)(uintptr_t)ngx_completion_pointer, ngx_completion_value, 0) == 0;
|
||||
ngx_completion_pointer = 0; /* one output kernel per evaluate: never fire on a later one */
|
||||
}
|
||||
if (result != CUDA_SUCCESS || trace_verbose())
|
||||
{
|
||||
tracef("cuLaunchKernel[%u] %s", sequence, find_function_name(function_handle));
|
||||
@@ -2569,6 +2681,7 @@ CUresult WINAPI cuMemcpyHtoDAsync(CUdeviceptr destination, const void* source, s
|
||||
{
|
||||
CUMEMCPYHTODASYNC_FN function = (CUMEMCPYHTODASYNC_FN)find_zluda_symbol("cuMemcpyHtoDAsync_v2");
|
||||
if (microcuda_k_write) microcuda_k_write(destination, bytes);
|
||||
stream = d4r_ngx_stream(stream);
|
||||
CUresult result = function != NULL ? function(destination, source, bytes, stream) : missing("cuMemcpyHtoDAsync_v2");
|
||||
TRACE_CALL(result, "cuMemcpyHtoDAsync->v2 dst=0x%llx src=%p bytes=%zu stream=%p result=%d",
|
||||
(unsigned long long)destination, source, bytes, stream, result);
|
||||
@@ -2906,7 +3019,6 @@ CUresult WINAPI cuTexObjectGetResourceDesc(void* descriptor, CUtexObject object)
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
/* Direct pass-throughs for driver APIs whose arguments are all integers or
|
||||
pointers, so the Windows x64 call forwards unchanged to ZLUDA (SysV). Unversioned
|
||||
names resolve to the versions the CUDA 12 driver returns for them. */
|
||||
@@ -2935,8 +3047,8 @@ static int d4r_forward_is_teardown(const char* target)
|
||||
D4R_FORWARD(cuEventCreate, cuEventCreate, 2, (D4rArg a0, D4rArg a1), (a0, a1))
|
||||
D4R_FORWARD(cuEventDestroy, cuEventDestroy_v2, 1, (D4rArg a0), (a0))
|
||||
D4R_FORWARD(cuEventDestroy_v2, cuEventDestroy_v2, 1, (D4rArg a0), (a0))
|
||||
D4R_FORWARD(cuEventRecord, cuEventRecord, 2, (D4rArg a0, D4rArg a1), (a0, a1))
|
||||
D4R_FORWARD(cuEventRecordWithFlags, cuEventRecordWithFlags, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuEventRecord, cuEventRecord, 2, (D4rArg a0, D4rArg a1), (a0, (D4rArg)d4r_ngx_stream((CUstream)a1)))
|
||||
D4R_FORWARD(cuEventRecordWithFlags, cuEventRecordWithFlags, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, (D4rArg)d4r_ngx_stream((CUstream)a1), a2))
|
||||
CUresult WINAPI cuEventSynchronize(D4rArg a0)
|
||||
{
|
||||
if (before_teardown())
|
||||
@@ -2995,9 +3107,9 @@ D4R_FORWARD(cuStreamCreate, cuStreamCreate, 2, (D4rArg a0, D4rArg a1), (a0, a1))
|
||||
D4R_FORWARD(cuStreamCreateWithPriority, cuStreamCreateWithPriority, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuStreamDestroy, cuStreamDestroy_v2, 1, (D4rArg a0), (a0))
|
||||
D4R_FORWARD(cuStreamDestroy_v2, cuStreamDestroy_v2, 1, (D4rArg a0), (a0))
|
||||
D4R_FORWARD(cuStreamSynchronize, cuStreamSynchronize, 1, (D4rArg a0), (a0))
|
||||
D4R_FORWARD(cuStreamQuery, cuStreamQuery, 1, (D4rArg a0), (a0))
|
||||
D4R_FORWARD(cuStreamWaitEvent, cuStreamWaitEvent, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuStreamSynchronize, cuStreamSynchronize, 1, (D4rArg a0), ((D4rArg)d4r_ngx_stream((CUstream)a0)))
|
||||
D4R_FORWARD(cuStreamQuery, cuStreamQuery, 1, (D4rArg a0), ((D4rArg)d4r_ngx_stream((CUstream)a0)))
|
||||
D4R_FORWARD(cuStreamWaitEvent, cuStreamWaitEvent, 3, (D4rArg a0, D4rArg a1, D4rArg a2), ((D4rArg)d4r_ngx_stream((CUstream)a0), a1, a2))
|
||||
D4R_FORWARD(cuStreamGetFlags, cuStreamGetFlags, 2, (D4rArg a0, D4rArg a1), (a0, a1))
|
||||
D4R_FORWARD(cuStreamGetPriority, cuStreamGetPriority, 2, (D4rArg a0, D4rArg a1), (a0, a1))
|
||||
D4R_FORWARD(cuCtxGetCurrent, cuCtxGetCurrent, 1, (D4rArg a0), (a0))
|
||||
@@ -3014,14 +3126,14 @@ D4R_FORWARD(cuMemGetInfo_v2, cuMemGetInfo_v2, 2, (D4rArg a0, D4rArg a1), (a0, a1
|
||||
D4R_FORWARD(cuMemcpyHtoD, cuMemcpyHtoD_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuMemcpyHtoD_v2, cuMemcpyHtoD_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuMemcpyDtoD, cuMemcpyDtoD_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuMemcpyDtoDAsync, cuMemcpyDtoDAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
|
||||
D4R_FORWARD(cuMemcpyDtoHAsync, cuMemcpyDtoHAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
|
||||
D4R_FORWARD(cuMemcpyDtoDAsync, cuMemcpyDtoDAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
|
||||
D4R_FORWARD(cuMemcpyDtoHAsync, cuMemcpyDtoHAsync_v2, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
|
||||
D4R_FORWARD(cuMemsetD8, cuMemsetD8_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuMemsetD16, cuMemsetD16_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuMemsetD32, cuMemsetD32_v2, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
D4R_FORWARD(cuMemsetD8Async, cuMemsetD8Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
|
||||
D4R_FORWARD(cuMemsetD16Async, cuMemsetD16Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
|
||||
D4R_FORWARD(cuMemsetD32Async, cuMemsetD32Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, a3))
|
||||
D4R_FORWARD(cuMemsetD8Async, cuMemsetD8Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
|
||||
D4R_FORWARD(cuMemsetD16Async, cuMemsetD16Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
|
||||
D4R_FORWARD(cuMemsetD32Async, cuMemsetD32Async, 4, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3), (a0, a1, a2, (D4rArg)d4r_ngx_stream((CUstream)a3)))
|
||||
D4R_FORWARD(cuMemsetD2D8, cuMemsetD2D8_v2, 5, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3, D4rArg a4), (a0, a1, a2, a3, a4))
|
||||
D4R_FORWARD(cuMemsetD2D32, cuMemsetD2D32_v2, 5, (D4rArg a0, D4rArg a1, D4rArg a2, D4rArg a3, D4rArg a4), (a0, a1, a2, a3, a4))
|
||||
D4R_FORWARD(cuFuncSetAttribute, cuFuncSetAttribute, 3, (D4rArg a0, D4rArg a1, D4rArg a2), (a0, a1, a2))
|
||||
@@ -3116,6 +3228,59 @@ static void* hip_symbol(const char* name)
|
||||
return hip != NULL ? dlsym(hip, name) : NULL;
|
||||
}
|
||||
|
||||
static CUstream ngx_priority_stream;
|
||||
static pthread_once_t ngx_priority_once = PTHREAD_ONCE_INIT;
|
||||
|
||||
static void create_ngx_priority_stream(void)
|
||||
{
|
||||
const char* setting = getenv("D4R_NGX_STREAM_PRIORITY");
|
||||
const int high = setting != NULL && strcmp(setting, "high") == 0;
|
||||
if (!high && (setting == NULL || strcmp(setting, "normal") != 0))
|
||||
return;
|
||||
typedef int (*STREAM_PRIORITY_RANGE_FN)(int*, int*);
|
||||
typedef int (*STREAM_CREATE_PRIORITY_FN)(void**, unsigned int, int);
|
||||
STREAM_PRIORITY_RANGE_FN range = (STREAM_PRIORITY_RANGE_FN)hip_symbol("hipDeviceGetStreamPriorityRange");
|
||||
STREAM_CREATE_PRIORITY_FN create = (STREAM_CREATE_PRIORITY_FN)hip_symbol("hipStreamCreateWithPriority");
|
||||
int least = 0, greatest = 0;
|
||||
ensure_context();
|
||||
if (context_setup_result != CUDA_SUCCESS || range == NULL || create == NULL || range(&least, &greatest) != 0)
|
||||
{
|
||||
tracef("D4R_NGX_STREAM_PRIORITY=%s: HIP stream priorities unavailable; NGX stays on the null stream", setting);
|
||||
return;
|
||||
}
|
||||
void* stream = NULL;
|
||||
const int priority = high ? greatest : 0;
|
||||
const int result = create(&stream, 0 /* hipStreamDefault: blocking */, priority);
|
||||
tracef("D4R_NGX_STREAM_PRIORITY=%s: NGX stream %p priority %d (range %d..%d) result=%d", setting, stream,
|
||||
priority, least, greatest, result);
|
||||
if (result == 0)
|
||||
ngx_priority_stream = (CUstream)stream;
|
||||
}
|
||||
|
||||
static CUstream d4r_ngx_stream(CUstream stream)
|
||||
{
|
||||
if (stream != NULL)
|
||||
return stream;
|
||||
pthread_once(&ngx_priority_once, create_ngx_priority_stream);
|
||||
return ngx_priority_stream; /* NULL unless D4R_NGX_STREAM_PRIORITY created one */
|
||||
}
|
||||
|
||||
CUresult WINAPI d4rNgxStreamPriority(int* priority)
|
||||
{
|
||||
/* For the shim's log: 1 when NGX runs on the substituted stream, with its HIP priority. */
|
||||
typedef int (*STREAM_GET_PRIORITY_FN)(void*, int*);
|
||||
CUstream stream = d4r_ngx_stream(NULL);
|
||||
STREAM_GET_PRIORITY_FN get = (STREAM_GET_PRIORITY_FN)hip_symbol("hipStreamGetPriority");
|
||||
if (priority == NULL)
|
||||
return CUDA_ERROR_INVALID_VALUE;
|
||||
*priority = 0;
|
||||
if (stream == NULL)
|
||||
return CUDA_ERROR_NOT_SUPPORTED;
|
||||
if (get != NULL)
|
||||
get(stream, priority);
|
||||
return CUDA_SUCCESS;
|
||||
}
|
||||
|
||||
CUresult WINAPI d4rImportVulkanMemory(void* client_device, uint64_t client_memory, uint64_t bytes, CUdeviceptr* pointer,
|
||||
void** memory)
|
||||
{
|
||||
@@ -3282,6 +3447,146 @@ typedef int (*VK_SIGNAL_SEMAPHORE_FN)(void*, const D4rVkSemaphoreSignalInfo*);
|
||||
typedef int (*VK_GET_SEMAPHORE_COUNTER_FN)(void*, uint64_t, uint64_t*);
|
||||
typedef int (*HIP_LAUNCH_HOST_FN)(void*, void (*)(void*), void*);
|
||||
|
||||
/* NGX's null-stream completion, independent of Wine scheduling/event polling. CPU cancellations
|
||||
and native callbacks share one timeline lock so a late callback never signals a lower value.
|
||||
No HIP or Wine APIs may run from the callback. The shim queues this only for the completed-prefix
|
||||
head, after every output write; resource retirement waits for its native completion condition. */
|
||||
typedef struct
|
||||
{
|
||||
void* device;
|
||||
uint64_t semaphore, value, pending;
|
||||
VK_SIGNAL_SEMAPHORE_FN signal;
|
||||
HIP_LAUNCH_HOST_FN launch;
|
||||
pthread_mutex_t lock;
|
||||
pthread_cond_t completed;
|
||||
int busy, cancelled, detached;
|
||||
} D4rNgxTimeline;
|
||||
|
||||
int WINAPI d4rNgxTimelineSignal(void* handle, uint64_t value)
|
||||
{
|
||||
D4rNgxTimeline* timeline = handle;
|
||||
pthread_mutex_lock(&timeline->lock);
|
||||
int result = 0;
|
||||
if (value > timeline->value)
|
||||
{
|
||||
D4rVkSemaphoreSignalInfo info = {1000207005, NULL, timeline->semaphore, value};
|
||||
result = timeline->signal(timeline->device, &info);
|
||||
if (result == 0) timeline->value = value;
|
||||
}
|
||||
pthread_mutex_unlock(&timeline->lock);
|
||||
return result;
|
||||
}
|
||||
|
||||
static void d4r_ngx_completed(void* data)
|
||||
{
|
||||
D4rNgxTimeline* timeline = data;
|
||||
pthread_mutex_lock(&timeline->lock);
|
||||
if (!__atomic_load_n(&timeline->cancelled, __ATOMIC_ACQUIRE) && timeline->pending > timeline->value)
|
||||
{
|
||||
D4rVkSemaphoreSignalInfo info = {1000207005, NULL, timeline->semaphore, timeline->pending};
|
||||
if (timeline->signal(timeline->device, &info) == 0) timeline->value = timeline->pending;
|
||||
/* A failed signal leaves the worker's normal retirement to retry. */
|
||||
}
|
||||
timeline->busy = 0;
|
||||
pthread_cond_signal(&timeline->completed);
|
||||
const int detached = timeline->detached;
|
||||
pthread_mutex_unlock(&timeline->lock);
|
||||
if (detached) { pthread_cond_destroy(&timeline->completed); pthread_mutex_destroy(&timeline->lock); free(timeline); }
|
||||
}
|
||||
|
||||
CUresult WINAPI d4rNgxTimelineCreate(void* client_device, uint64_t client_semaphore, void** handle)
|
||||
{
|
||||
if (client_device == NULL || client_semaphore == 0 || handle == NULL) return CUDA_ERROR_INVALID_VALUE;
|
||||
const uint64_t* device_object = (const uint64_t*)(uintptr_t)((const uint64_t*)client_device)[1];
|
||||
const uint64_t* semaphore_object = (const uint64_t*)(uintptr_t)client_semaphore;
|
||||
if (device_object == NULL || device_object[1] != (uint64_t)(uintptr_t)client_device ||
|
||||
semaphore_object[1] != client_semaphore) return CUDA_ERROR_NOT_SUPPORTED;
|
||||
void* vulkan = dlopen("libvulkan.so.1", RTLD_NOW | RTLD_NOLOAD);
|
||||
VK_GET_DEVICE_PROC_ADDR_FN get = vulkan != NULL ? dlsym(vulkan, "vkGetDeviceProcAddr") : NULL;
|
||||
HIP_LAUNCH_HOST_FN launch = (HIP_LAUNCH_HOST_FN)hip_symbol("hipLaunchHostFunc");
|
||||
if (get == NULL || launch == NULL) return CUDA_ERROR_NOT_SUPPORTED;
|
||||
void* device = (void*)(uintptr_t)device_object[0];
|
||||
VK_SIGNAL_SEMAPHORE_FN signal = (VK_SIGNAL_SEMAPHORE_FN)get(device, "vkSignalSemaphore");
|
||||
VK_GET_SEMAPHORE_COUNTER_FN counter = (VK_GET_SEMAPHORE_COUNTER_FN)get(device, "vkGetSemaphoreCounterValue");
|
||||
uint64_t value = 0;
|
||||
if (signal == NULL || counter == NULL || counter(device, semaphore_object[0], &value) != 0)
|
||||
return CUDA_ERROR_NOT_SUPPORTED;
|
||||
D4rNgxTimeline* timeline = calloc(1, sizeof(*timeline));
|
||||
if (timeline == NULL) return CUDA_ERROR_OUT_OF_MEMORY;
|
||||
if (pthread_mutex_init(&timeline->lock, NULL) != 0) { free(timeline); return CUDA_ERROR_UNKNOWN; }
|
||||
pthread_condattr_t attr;
|
||||
if (pthread_condattr_init(&attr) != 0) { pthread_mutex_destroy(&timeline->lock); free(timeline); return CUDA_ERROR_UNKNOWN; }
|
||||
const int condition = pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) != 0 ||
|
||||
pthread_cond_init(&timeline->completed, &attr) != 0;
|
||||
pthread_condattr_destroy(&attr);
|
||||
if (condition) { pthread_mutex_destroy(&timeline->lock); free(timeline); return CUDA_ERROR_UNKNOWN; }
|
||||
timeline->device = device; timeline->semaphore = semaphore_object[0]; timeline->value = value;
|
||||
timeline->signal = signal; timeline->launch = launch; *handle = timeline;
|
||||
return CUDA_SUCCESS;
|
||||
}
|
||||
|
||||
CUresult WINAPI d4rNgxTimelinePrepare(void* handle)
|
||||
{
|
||||
D4rNgxTimeline* timeline = handle;
|
||||
pthread_mutex_lock(&timeline->lock);
|
||||
const int busy = timeline->busy;
|
||||
if (!busy) __atomic_store_n(&timeline->cancelled, 0, __ATOMIC_RELEASE);
|
||||
pthread_mutex_unlock(&timeline->lock);
|
||||
return busy ? CUDA_ERROR_NOT_READY : CUDA_SUCCESS;
|
||||
}
|
||||
|
||||
void WINAPI d4rNgxTimelineCancel(void* handle)
|
||||
{
|
||||
/* Called by the independent input watchdog before its coherent release store: no queue calls/locks. */
|
||||
__atomic_store_n(&((D4rNgxTimeline*)handle)->cancelled, 1, __ATOMIC_RELEASE);
|
||||
}
|
||||
|
||||
CUresult WINAPI d4rNgxTimelineQueue(void* handle, uint64_t value)
|
||||
{
|
||||
D4rNgxTimeline* timeline = handle;
|
||||
pthread_mutex_lock(&timeline->lock);
|
||||
if (timeline->busy) { pthread_mutex_unlock(&timeline->lock); return CUDA_ERROR_NOT_READY; }
|
||||
timeline->pending = value; timeline->busy = 1;
|
||||
pthread_mutex_unlock(&timeline->lock);
|
||||
const int result = timeline->launch(d4r_ngx_stream(NULL), d4r_ngx_completed, timeline);
|
||||
if (result != 0)
|
||||
{
|
||||
pthread_mutex_lock(&timeline->lock); timeline->busy = 0; pthread_mutex_unlock(&timeline->lock);
|
||||
}
|
||||
return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_NOT_SUPPORTED;
|
||||
}
|
||||
|
||||
void WINAPI d4rNgxTimelineDestroy(void* handle)
|
||||
{
|
||||
D4rNgxTimeline* timeline = handle;
|
||||
pthread_mutex_lock(&timeline->lock);
|
||||
__atomic_store_n(&timeline->cancelled, 1, __ATOMIC_RELEASE);
|
||||
const int busy = timeline->busy;
|
||||
timeline->detached = busy;
|
||||
pthread_mutex_unlock(&timeline->lock);
|
||||
/* Normally the native condition drained the callback. If a device error prevented that drain, retain
|
||||
only this self-owned ticket until its cancelled callback returns, never a pointer into Feature. */
|
||||
if (!busy) { pthread_cond_destroy(&timeline->completed); pthread_mutex_destroy(&timeline->lock); free(timeline); }
|
||||
}
|
||||
|
||||
CUresult WINAPI d4rNgxTimelineWait(void* handle)
|
||||
{
|
||||
D4rNgxTimeline* timeline = handle;
|
||||
struct timespec deadline;
|
||||
if (clock_gettime(CLOCK_MONOTONIC, &deadline) != 0) return CUDA_ERROR_UNKNOWN;
|
||||
deadline.tv_sec += 5;
|
||||
pthread_mutex_lock(&timeline->lock);
|
||||
int result = 0;
|
||||
while (timeline->busy && result == 0)
|
||||
result = pthread_cond_timedwait(&timeline->completed, &timeline->lock, &deadline);
|
||||
const int busy = timeline->busy;
|
||||
pthread_mutex_unlock(&timeline->lock);
|
||||
/* The callback runs after every output write on the null stream. Its condition is sufficient to
|
||||
retire those resources, without polling/recording another GPU event. A timeout keeps the ticket
|
||||
owned; the caller cancels, disables this mode and falls back to its normal output synchronization. */
|
||||
return !busy ? CUDA_SUCCESS : result == ETIMEDOUT ? CUDA_ERROR_NOT_READY : CUDA_ERROR_UNKNOWN;
|
||||
}
|
||||
|
||||
typedef struct D4rEngineNet
|
||||
{
|
||||
int k_preset; /* D4RKHIP1: the token grid comes from the output dimensions and the cycle has eight graphs */
|
||||
@@ -3333,7 +3638,6 @@ typedef int (*HIP_CAPTURE_BEGIN_FN)(void*, int);
|
||||
typedef int (*HIP_CAPTURE_END_FN)(void*, void**);
|
||||
typedef int (*HIP_GRAPH_INSTANTIATE_FN)(void**, void*, void**, char*, size_t);
|
||||
|
||||
|
||||
CUresult WINAPI d4rEngineNetReleaseInput(void* handle, uint32_t value)
|
||||
{
|
||||
D4rEngineNet* net = (D4rEngineNet*)handle;
|
||||
@@ -4175,7 +4479,7 @@ CUresult WINAPI d4rMemcpy2DAsync(const D4rMemcpy2D* copy, CUstream stream)
|
||||
D4rMemcpy2D hip = *copy;
|
||||
hip.srcMemoryType = (unsigned int)src;
|
||||
hip.dstMemoryType = (unsigned int)dst;
|
||||
const int result = function(&hip, NULL);
|
||||
const int result = function(&hip, d4r_ngx_stream(NULL));
|
||||
if (result != 0 || trace_verbose())
|
||||
tracef("d4rMemcpy2DAsync %zux%zu result=%d", copy->WidthInBytes, copy->Height, result);
|
||||
return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_INVALID_VALUE;
|
||||
@@ -4194,12 +4498,68 @@ CUresult WINAPI d4rStreamWaitValue32(CUdeviceptr pointer, uint32_t value)
|
||||
function = (HIP_STREAM_WAIT_VALUE32_FN)hip_symbol("hipStreamWaitValue32");
|
||||
if (function == NULL)
|
||||
return CUDA_ERROR_NOT_SUPPORTED;
|
||||
const int result = function(NULL, (void*)(uintptr_t)pointer, value, 0 /* hipStreamWaitValueGte */, 0xffffffffu);
|
||||
const int result = function(d4r_ngx_stream(NULL), (void*)(uintptr_t)pointer, value, 0 /* hipStreamWaitValueGte */,
|
||||
0xffffffffu);
|
||||
if (result != 0 || trace_verbose())
|
||||
tracef("d4rStreamWaitValue32 %p >= %u result=%d", (void*)(uintptr_t)pointer, value, result);
|
||||
return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_INVALID_VALUE;
|
||||
}
|
||||
|
||||
/* Exact per-slot tags, rather than the engine's monotonic Gte word: an old or later frame must never
|
||||
satisfy NGX's wait, including across u32 wrap. ZLUDA uses HIP's null stream for null CUstream kernel
|
||||
launches; d4rMemcpy2DAsync uses that same stream. This export therefore orders both input copies and
|
||||
NGX kernels without adding an unsupported CUDA driver entry point to ZLUDA. The caller owns a coherent
|
||||
CPU mapping and arms its watchdog before entering here, so a blocked HIP call can still be released. */
|
||||
CUresult WINAPI d4rStreamWaitValue32Eq(CUdeviceptr pointer, uint32_t value)
|
||||
{
|
||||
typedef int (*GET_ATTRIBUTE_FN)(int*, int, int);
|
||||
typedef int (*GET_DEVICE_FN)(int*);
|
||||
ensure_context();
|
||||
if (context_setup_result != CUDA_SUCCESS)
|
||||
return context_setup_result;
|
||||
HIP_STREAM_WAIT_VALUE32_FN wait = (HIP_STREAM_WAIT_VALUE32_FN)hip_symbol("hipStreamWaitValue32");
|
||||
GET_ATTRIBUTE_FN attribute = (GET_ATTRIBUTE_FN)hip_symbol("hipDeviceGetAttribute");
|
||||
GET_DEVICE_FN device = (GET_DEVICE_FN)hip_symbol("hipGetDevice");
|
||||
int current = 0, supported = 0;
|
||||
if (pointer == 0 || wait == NULL || attribute == NULL || device == NULL || device(¤t) != 0 ||
|
||||
attribute(&supported, 10013 /* hipDeviceAttributeCanUseStreamWaitValue */, current) != 0 || !supported)
|
||||
return CUDA_ERROR_NOT_SUPPORTED;
|
||||
const int result = wait(d4r_ngx_stream(NULL), (void*)(uintptr_t)pointer, value, 1 /* hipStreamWaitValueEq */,
|
||||
0xffffffffu);
|
||||
if (result != 0 || trace_verbose())
|
||||
tracef("d4rStreamWaitValue32Eq %p == %u result=%d", (void*)(uintptr_t)pointer, value, result);
|
||||
return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_NOT_SUPPORTED; /* failure: no wait queued */
|
||||
}
|
||||
|
||||
double WINAPI d4rThreadCpuTimeMs(void)
|
||||
{
|
||||
struct timespec resolution, now;
|
||||
if (clock_getres(CLOCK_THREAD_CPUTIME_ID, &resolution) != 0 || resolution.tv_sec != 0 ||
|
||||
resolution.tv_nsec > 100000 /* 100 us: too coarse for these sub-ms submission/wait costs */ ||
|
||||
clock_gettime(CLOCK_THREAD_CPUTIME_ID, &now) != 0)
|
||||
return -1.0;
|
||||
return (double)now.tv_sec * 1000.0 + (double)now.tv_nsec / 1000000.0;
|
||||
}
|
||||
|
||||
/* D4R_SHIM_GPU_COMPLETION: queues a 32-bit write of `value` to `pointer` on the stream NGX runs on, behind the
|
||||
frame's output writes. The game's queue polls that word (kernels/support/vulkan_wait.comp) instead of waiting
|
||||
for a CPU-signalled timeline value, so DLSS's completion reaches the game without a host round trip. */
|
||||
CUresult WINAPI d4rNgxWriteValue32(CUdeviceptr pointer, uint32_t value)
|
||||
{
|
||||
static HIP_STREAM_WRITE_VALUE32_FN write;
|
||||
ensure_context();
|
||||
if (context_setup_result != CUDA_SUCCESS)
|
||||
return context_setup_result;
|
||||
if (write == NULL)
|
||||
write = (HIP_STREAM_WRITE_VALUE32_FN)hip_symbol("hipStreamWriteValue32");
|
||||
if (write == NULL || pointer == 0)
|
||||
return CUDA_ERROR_NOT_SUPPORTED;
|
||||
const int result = write(d4r_ngx_stream(NULL), (void*)(uintptr_t)pointer, value, 0);
|
||||
if (result != 0 || trace_verbose())
|
||||
tracef("d4rNgxWriteValue32 %p = %u result=%d", (void*)(uintptr_t)pointer, value, result);
|
||||
return result == 0 ? CUDA_SUCCESS : CUDA_ERROR_NOT_SUPPORTED;
|
||||
}
|
||||
|
||||
CUresult WINAPI d4rWriteValue32(CUdeviceptr pointer, uint32_t value)
|
||||
{
|
||||
static HIP_STREAM_WRITE_VALUE32_FN write;
|
||||
|
||||
@@ -104,10 +104,24 @@
|
||||
@ stdcall d4rSetShutdownHook(ptr) d4rSetShutdownHook
|
||||
@ stdcall d4rEventSynchronize(ptr) d4rEventSynchronize
|
||||
@ stdcall d4rStreamWaitValue32(int64 long) d4rStreamWaitValue32
|
||||
@ stdcall d4rStreamWaitValue32Eq(int64 long) d4rStreamWaitValue32Eq
|
||||
@ stdcall d4rNgxStreamPriority(ptr) d4rNgxStreamPriority
|
||||
@ stdcall d4rNgxWriteValue32(int64 long) d4rNgxWriteValue32
|
||||
@ stdcall d4rNgxArmCompletion(int64 long long) d4rNgxArmCompletion
|
||||
@ stdcall d4rNgxCompletionFired() d4rNgxCompletionFired
|
||||
@ stdcall d4rThreadCpuTimeMs() d4rThreadCpuTimeMs
|
||||
@ stdcall d4rWriteValue32(int64 long) d4rWriteValue32
|
||||
@ stdcall d4rSetArrayRedirect(ptr int64 long) d4rSetArrayRedirect
|
||||
@ stdcall d4rRegisterLinearTexture(int64 int64 int64 long long) d4rRegisterLinearTexture
|
||||
@ stdcall d4rSetEnv(str str long) d4rSetEnv
|
||||
@ stdcall d4rLoadError() d4rLoadError
|
||||
@ stdcall d4rOutputKernelNative() d4rOutputKernelNative
|
||||
@ stdcall d4rOutputKernelRgba8() d4rOutputKernelRgba8
|
||||
@ stdcall d4rNgxTimelineCreate(ptr int64 ptr) d4rNgxTimelineCreate
|
||||
@ stdcall d4rNgxTimelinePrepare(ptr) d4rNgxTimelinePrepare
|
||||
@ stdcall d4rNgxTimelineCancel(ptr) d4rNgxTimelineCancel
|
||||
@ stdcall d4rNgxTimelineQueue(ptr int64) d4rNgxTimelineQueue
|
||||
@ stdcall d4rNgxTimelineSignal(ptr int64) d4rNgxTimelineSignal
|
||||
@ stdcall d4rNgxTimelineDestroy(ptr) d4rNgxTimelineDestroy
|
||||
@ stdcall d4rNgxTimelineWait(ptr) d4rNgxTimelineWait
|
||||
@ stdcall d4rDilateMotionVectors(ptr) d4rDilateMotionVectors
|
||||
Reference in new issue
Block a user