TL;DR: Add an experimental FSR Ray Regeneration backend on Windows, run the K

network natively on HIP with Vulkan->HIP shared memory, extend K kernels to
gfx11 (RDNA3), and add dependency/packaging/test tooling around them.

Windows: FSR RR backend, native HIP K runner, gfx11 K kernels, tooling

FSR Ray Regeneration (experimental)
- Add fsr_rr_backend.{h,cpp}, fsr_rr_passes.h and the fsr_rr/ prepare and
  compose compute shaders. They run AMD's FidelityFX denoiser
  (amd_fidelityfx_denoiser_dx12.dll, SDK 2.3, not bundled) on a private
  vkd3d-proton device.
- Add the amdxc64 shim (tools/windows/amdxc64_shim) and two vkd3d-proton
  patches for libcxx includes. AMD's amdxc64.dll is never loaded.
- d4r_nvngx_windows: RrFeature now selects between the native DLSS RR
  backend and the FSR backend (D4R_RR_BACKEND=fsr). After the denoiser it
  runs an internal Super Sampling stage (feature 1) to upscale. If SR is
  unavailable and output != render size, the frame fails; it is never
  returned noisy.
- d4r.ini / player_config: new [RayReconstruction] FsrDenoiser,
  FsrStabilityBias (auto or >= 0) and FsrDetail (0-1), with validation, and
  matching D4R_FSR_* environment variables.
- Add the FSR RR fixture, output-check and ngx test scripts, and a shader
  compile script.

Native HIP K runner (Windows)
- CMake: add d4r_engine_k_hip (d4r-k-hip) plus separate volk/engine copies,
  because VK_USE_PLATFORM_WIN32_KHR changes volk's entry points.
- Share Vulkan memory with HIP via Win32 opaque handles on Windows
  (VK_KHR_external_memory_win32) and via fds on Linux. Handle types are now
  intptr_t, and the platform choice is in D4R_EXT_MEMORY_* macros.
- HipNetwork: add debugRepeatHashes and debugNodeMicros, exposed as
  D4R_HIP_REPEAT_CHECK, D4R_HIP_REPEAT_ALL and D4R_HIP_NODE_PROFILE in
  replay.cpp.
- replay.cpp: display-resolution-motion frames (FrameFlagDisplayMotion) no
  longer require near2rnd == out2rnd. Both values must still be in (0, 1].

K kernels on gfx11
- Allow position-token and enc1 tiling on __GFX11__ as well as gfx12.
- Add an opt-in ds_bpermute cross-half exchange
  (D4R_WINDOWS_GFX11_DS_EXCHANGE, enabled for K on Windows gfx11). M keeps
  its LDS path.
- Fix a block-barrier race in the position layer. Without POS_VSHARE a fast
  wave could overwrite its hb rows while a slower wave still read them for a
  later head, giving rare wrong results on gfx1101 (enc0 outputs differing
  between identical launches).

Packaging and install
- install.bat, d4r.ini, WINDOWS_README: FSR option, and
  install_dependencies.py to install pinned NVIDIA dependencies with staged,
  hash-verified replacement and rollback.
- Update the release package, local package refresh and test scripts.
- Add the InteropBatchInputCopies option (default false). Batching is faster
  on some synthetic runs, but player temporal exactness is unresolved.

Tooling and tests
- K model: capture_k_weights, build_k_model, compare_k_outputs,
  benchmark_k_hip, build_capture_runtime, test_k_model and the
  zluda-k-output-capture patch. Captured weights and models are private
  local assets and must not enter release archives.
- Synthetic probes: test_m_ngx, test_sr_dynamic,
  prepare_rr_dynamic_fixture and the d3d12/rr_ngx probe updates.
- Add tests/windows/test_install_dependencies.py.

Docs
- Update the windows-fsr-rr, k-performance, local-package, rr-interop and
  rr-model-cache docs. Add windows-parity-handoff.md and
  windows-parity-continuation-prompt.md.
This commit is contained in:
ayo committed 2026-10-17 01:43:06 -04:00
1 parent 247fe1092d
commit a617c818de
62 files changed
+5894 -89

No files matched your search

+49 -4
View File
@@ -9,6 +9,7 @@
#include "hip_net.h"
#include <hip/hip_runtime_api.h>
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <stdexcept>
@@ -64,8 +65,13 @@ uint32_t global32(hipModule_t module, const char* name, uint32_t fallback) {
}
return v;
}
void* import(int fd, uint64_t bytes, hipExternalMemory_t& memory) {
hipExternalMemoryHandleDesc desc{}; desc.type = hipExternalMemoryHandleTypeOpaqueFd; desc.handle.fd = fd; desc.size = bytes;
void* import(intptr_t fd, uint64_t bytes, hipExternalMemory_t& memory) {
hipExternalMemoryHandleDesc desc{}; desc.size = bytes;
#ifdef _WIN32
desc.type = hipExternalMemoryHandleTypeOpaqueWin32; desc.handle.win32.handle = reinterpret_cast<void*>(fd);
#else
desc.type = hipExternalMemoryHandleTypeOpaqueFd; desc.handle.fd = int(fd);
#endif
HIPCHECK(hipImportExternalMemory(&memory, &desc));
hipExternalMemoryBufferDesc buffer{}; buffer.offset = 0; buffer.size = bytes;
void* at = nullptr; HIPCHECK(hipExternalMemoryGetMappedBuffer(&at, memory, &buffer));
@@ -101,6 +107,7 @@ struct HipNetwork::Impl {
std::vector<Node> preps;
std::vector<Node> nodes[MaxGraphs];
std::vector<void*> owned;
std::vector<uint64_t> ownedBytes;
hipExternalMemory_t shared[2]{};
hipStream_t stream = nullptr;
hipGraph_t graph[MaxGraphs]{};
@@ -108,6 +115,7 @@ struct HipNetwork::Impl {
uint32_t graphs = 2, frame = 0;
hipEvent_t start = nullptr, stop = nullptr;
bool pending = false;
void* outputPtr = nullptr; uint64_t outputBytes = 0;
~Impl() {
if (stream) (void)hipStreamSynchronize(stream);
for (auto e : executable) if (e) (void)hipGraphExecDestroy(e);
@@ -125,7 +133,7 @@ struct HipNetwork::Impl {
}
};
HipNetwork::HipNetwork(const std::string& directory, uint32_t width, uint32_t height, int tokensFd, int outputFd,
HipNetwork::HipNetwork(const std::string& directory, uint32_t width, uint32_t height, intptr_t tokensFd, intptr_t outputFd,
uint64_t tokensBytes, uint64_t outputBytes) : impl(std::make_unique<Impl>()) {
auto& n = *impl;
HIPCHECK(hipInit(0));
@@ -143,6 +151,7 @@ HipNetwork::HipNetwork(const std::string& directory, uint32_t width, uint32_t he
if (count != layers || file.size() < 16 + 64ull * count) throw std::runtime_error("invalid hipnet.bin");
void* tokens = import(tokensFd, tokensBytes, n.shared[0]);
void* output = import(outputFd, outputBytes, n.shared[1]);
n.outputPtr = output; n.outputBytes = outputBytes;
// Both dimensions describe the layer-0 token grid: preset M's render dimensions, preset K's output
// dimensions (makeKShape: max 256, align 32 of the output quarter resolution).
uint32_t tw = 0, th = 0;
@@ -191,7 +200,7 @@ HipNetwork::HipNetwork(const std::string& directory, uint32_t width, uint32_t he
return uint32_t(names.size() - 1);
};
auto allocate = [&](uint64_t bytes) {
void* p = nullptr; HIPCHECK(hipMalloc(&p, size_t(bytes) + 4096)); n.owned.push_back(p); return p;
void* p = nullptr; HIPCHECK(hipMalloc(&p, size_t(bytes) + 4096)); n.owned.push_back(p); n.ownedBytes.push_back(bytes); return p;
};
std::vector<uint32_t> layerKernel(layers), divisor(layers);
std::vector<int32_t> shiftX(layers), shiftY(layers);
@@ -328,5 +337,41 @@ bool HipNetwork::done() {
check(e, "hipEventQuery"); n.pending = false; return true;
}
void HipNetwork::wait() { HIPCHECK(hipStreamSynchronize(impl->stream)); impl->pending = false; }
std::vector<uint64_t> HipNetwork::debugRepeatHashes(uint32_t rounds) {
// per launch: the output buffer hash, then one hash per owned allocation (weights and activations)
auto& n = *impl; std::vector<uint64_t> hashes; std::vector<unsigned char> host;
auto hash = [&](void* p, uint64_t bytes) {
host.resize(size_t(bytes)); HIPCHECK(hipMemcpyDtoH(host.data(), p, host.size()));
uint64_t h = 1469598103934665603ull;
// 64-bit words: byte-wise FNV over the ~18 MB head buffer cost more than the network itself
const size_t words = host.size() / 8; const uint64_t* data = reinterpret_cast<const uint64_t*>(host.data());
for (size_t i = 0; i < words; ++i) { h ^= data[i]; h *= 1099511628211ull; h ^= h >> 29; }
for (size_t i = words * 8; i < host.size(); ++i) { h ^= host[i]; h *= 1099511628211ull; }
hashes.push_back(h);
};
for (uint32_t r = 0; r < rounds; ++r)
for (uint32_t i = 0; i < 8; ++i) {
launch(); wait();
hash(n.outputPtr, n.outputBytes);
if (std::getenv("D4R_HIP_REPEAT_ALL"))
for (size_t k = 0; k < n.owned.size(); ++k) hash(n.owned[k], n.ownedBytes[k]);
}
return hashes;
}
std::vector<float> HipNetwork::debugNodeMicros(uint32_t phase, uint32_t trials) {
auto& n = *impl; const auto& nodes = n.nodes[phase % n.graphs];
std::vector<std::vector<float>> samples(nodes.size());
hipEvent_t a = nullptr, b = nullptr; HIPCHECK(hipEventCreate(&a)); HIPCHECK(hipEventCreate(&b));
for (uint32_t t = 0; t < trials + 2; ++t)
for (size_t i = 0; i < nodes.size(); ++i) {
HIPCHECK(hipEventRecord(a, n.stream)); n.run(nodes[i]); HIPCHECK(hipEventRecord(b, n.stream));
HIPCHECK(hipEventSynchronize(b)); float ms = 0; HIPCHECK(hipEventElapsedTime(&ms, a, b));
if (t >= 2) samples[i].push_back(ms * 1000.0f);
}
(void)hipEventDestroy(a); (void)hipEventDestroy(b);
std::vector<float> result;
for (auto& s : samples) { std::sort(s.begin(), s.end()); result.push_back(s[s.size() / 2]); }
return result;
}
float HipNetwork::lastMilliseconds() { float ms = 0; HIPCHECK(hipEventElapsedTime(&ms, impl->start, impl->stop)); return ms; }
} // namespace d4r
+9 -2
View File
@@ -19,13 +19,14 @@
#include <cstdint>
#include <memory>
#include <string>
#include <vector>
namespace d4r {
class HipNetwork {
public:
// tokensFd, outputFd: opaque file descriptors of the two shared Vulkan allocations (ownership passes to HIP);
// tokensFd, outputFd: opaque file descriptors (Linux) or Win32 NT handles (Windows; the caller keeps ownership) of the two shared Vulkan allocations (ownership passes to HIP);
// allocationBytes: the size of each allocation; the buffers start at offset 0. width/height are preset
// dependent, as above.
HipNetwork(const std::string& modelDirectory, uint32_t width, uint32_t height, int tokensFd, int outputFd,
HipNetwork(const std::string& modelDirectory, uint32_t width, uint32_t height, intptr_t tokensFd, intptr_t outputFd,
uint64_t tokensAllocationBytes, uint64_t outputAllocationBytes);
~HipNetwork();
HipNetwork(const HipNetwork&) = delete;
@@ -34,6 +35,12 @@ public:
bool done(); // true once the queued layers have finished
void wait(); // blocks until they have
float lastMilliseconds(); // GPU time of the last finished launch
// Diagnostic: relaunches the network `rounds` x 8 times on the unchanged input tokens (the graphs cycle with
// period 8) and returns an FNV-1a hash of the output buffer after each launch, rounds-major.
std::vector<uint64_t> debugRepeatHashes(uint32_t rounds);
// Diagnostic: median GPU microseconds of every launch node of graph `phase` run individually (not graph-fused),
// over `trials` runs. Node order is network order, a layer's post kernels following its main kernel.
std::vector<float> debugNodeMicros(uint32_t phase, uint32_t trials);
private:
struct Impl;
std::unique_ptr<Impl> impl;
+49 -8
View File
@@ -6,6 +6,14 @@
#ifdef D4R_ENGINE_HIP
#include "hip_net.h"
#endif
// Opaque memory handle shared with HIP: a file descriptor on Linux, a Win32 NT handle on Windows.
#ifdef _WIN32
#define D4R_EXT_MEMORY_EXTENSION "VK_KHR_external_memory_win32"
#define D4R_EXT_MEMORY_HANDLE VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_BIT
#else
#define D4R_EXT_MEMORY_EXTENSION VK_KHR_EXTERNAL_MEMORY_FD_EXTENSION_NAME
#define D4R_EXT_MEMORY_HANDLE VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT
#endif
#include <algorithm>
#include <chrono>
#include <cstring>
@@ -87,8 +95,8 @@ struct Context {
throw std::runtime_error("VK_KHR_cooperative_matrix required");
if (fp8 && std::none_of(extensions.begin(), extensions.end(), [](const auto& e) { return !std::strcmp(e.extensionName, VK_EXT_SHADER_FLOAT8_EXTENSION_NAME); }))
throw std::runtime_error("VK_EXT_shader_float8 required for M");
if (externalMemory && std::none_of(extensions.begin(), extensions.end(), [](const auto& e) { return !std::strcmp(e.extensionName, VK_KHR_EXTERNAL_MEMORY_FD_EXTENSION_NAME); }))
throw std::runtime_error("VK_KHR_external_memory_fd required for the native HIP network");
if (externalMemory && std::none_of(extensions.begin(), extensions.end(), [](const auto& e) { return !std::strcmp(e.extensionName, D4R_EXT_MEMORY_EXTENSION); }))
throw std::runtime_error("external memory extension required for the native HIP network");
VkPhysicalDeviceShaderFloat8FeaturesEXT float8{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_FLOAT8_FEATURES_EXT};
VkPhysicalDeviceCooperativeMatrixFeaturesKHR matrix{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_COOPERATIVE_MATRIX_FEATURES_KHR, fp8 ? &float8 : nullptr};
VkPhysicalDeviceVulkan13Features f13{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VULKAN_1_3_FEATURES, &matrix};
@@ -137,7 +145,7 @@ struct Context {
const char* enabled[3] = {VK_KHR_COOPERATIVE_MATRIX_EXTENSION_NAME};
uint32_t enabledCount = 1;
if (fp8) enabled[enabledCount++] = VK_EXT_SHADER_FLOAT8_EXTENSION_NAME;
if (externalMemory) enabled[enabledCount++] = VK_KHR_EXTERNAL_MEMORY_FD_EXTENSION_NAME;
if (externalMemory) enabled[enabledCount++] = D4R_EXT_MEMORY_EXTENSION;
VkDeviceCreateInfo di{VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO, &features};
di.queueCreateInfoCount = 1; di.pQueueCreateInfos = &qi; di.enabledExtensionCount = enabledCount;
di.ppEnabledExtensionNames = enabled;
@@ -212,16 +220,16 @@ struct Buffer {
};
#ifdef D4R_ENGINE_HIP
// A storage buffer whose memory another API (HIP) imports through a file descriptor, for the native network.
struct Shared { VkBuffer buffer{}; VkDeviceMemory memory{}; VkDeviceSize allocation = 0; int fd = -1; };
struct Shared { VkBuffer buffer{}; VkDeviceMemory memory{}; VkDeviceSize allocation = 0; intptr_t fd = -1; };
static Shared shared(const Context& c, VkDeviceSize size) {
Shared s;
VkExternalMemoryBufferCreateInfo ei{VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO};
ei.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT;
ei.handleTypes = D4R_EXT_MEMORY_HANDLE;
VkBufferCreateInfo bi{VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, &ei}; bi.size = size; bi.usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT;
CK(vkCreateBuffer(c.device, &bi, nullptr, &s.buffer));
VkMemoryRequirements r; vkGetBufferMemoryRequirements(c.device, s.buffer, &r);
VkExportMemoryAllocateInfo xi{VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO};
xi.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT;
xi.handleTypes = D4R_EXT_MEMORY_HANDLE;
VkMemoryAllocateInfo ai{VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, &xi}; ai.allocationSize = s.allocation = r.size;
uint32_t index = c.memory.memoryTypeCount;
for (uint32_t i = 0; i < c.memory.memoryTypeCount; ++i)
@@ -229,11 +237,19 @@ static Shared shared(const Context& c, VkDeviceSize size) {
if (index == c.memory.memoryTypeCount) throw std::runtime_error("no device-local memory type");
ai.memoryTypeIndex = index;
CK(vkAllocateMemory(c.device, &ai, nullptr, &s.memory)); CK(vkBindBufferMemory(c.device, s.buffer, s.memory, 0));
#ifdef _WIN32
VkMemoryGetWin32HandleInfoKHR gi{VK_STRUCTURE_TYPE_MEMORY_GET_WIN32_HANDLE_INFO_KHR};
gi.memory = s.memory; gi.handleType = D4R_EXT_MEMORY_HANDLE;
const auto getHandle = reinterpret_cast<PFN_vkGetMemoryWin32HandleKHR>(vkGetDeviceProcAddr(c.device, "vkGetMemoryWin32HandleKHR"));
if (!getHandle) throw std::runtime_error("vkGetMemoryWin32HandleKHR is unavailable");
HANDLE handle = nullptr; CK(getHandle(c.device, &gi, &handle)); s.fd = reinterpret_cast<intptr_t>(handle);
#else
VkMemoryGetFdInfoKHR gi{VK_STRUCTURE_TYPE_MEMORY_GET_FD_INFO_KHR};
gi.memory = s.memory; gi.handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT;
gi.memory = s.memory; gi.handleType = D4R_EXT_MEMORY_HANDLE;
const auto getFd = reinterpret_cast<PFN_vkGetMemoryFdKHR>(vkGetDeviceProcAddr(c.device, "vkGetMemoryFdKHR"));
if (!getFd) throw std::runtime_error("vkGetMemoryFdKHR is unavailable");
CK(getFd(c.device, &gi, &s.fd));
int fd = -1; CK(getFd(c.device, &gi, &fd)); s.fd = fd;
#endif
return s;
}
#endif
@@ -419,6 +435,31 @@ int run(const std::string& modelDir, const std::string& caseFile, const std::str
context.submit();
#ifdef D4R_ENGINE_HIP
network->launch(); network->wait();
static bool nodesProfiled = false;
if (!nodesProfiled && std::getenv("D4R_HIP_NODE_PROFILE")) {
nodesProfiled = true; float total = 0;
auto micros = network->debugNodeMicros(0, 30);
for (size_t k = 0; k < micros.size(); ++k) { std::cout << " node " << k << ": " << micros[k] << " us\n"; total += micros[k]; }
std::cout << "hip node profile total (individually launched, phase 0): " << total << " us\n";
}
static bool repeatChecked = false;
if (!repeatChecked && std::getenv("D4R_HIP_REPEAT_CHECK")) {
repeatChecked = true;
const uint32_t rounds = uint32_t(std::atoi(std::getenv("D4R_HIP_REPEAT_CHECK")));
auto hashes = network->debugRepeatHashes(rounds); unsigned bad = 0;
const size_t stride = hashes.size() / (size_t(rounds) * 8); // 1 unless D4R_HIP_REPEAT_ALL
for (uint32_t phase = 0; phase < 8; ++phase)
for (uint32_t r = 1; r < rounds; ++r) {
bool any = false;
for (size_t k = 0; k < stride; ++k)
if (hashes[(size_t(r) * 8 + phase) * stride + k] != hashes[size_t(phase) * stride + k]) {
any = true; std::cout << " mismatch round " << r << " phase " << phase << " slot " << k << " (0=output, 1+=owned alloc)\n";
}
if (any) ++bad;
}
std::cout << "hip repeat check: " << rounds << " rounds x 8 phases, mismatching launches vs round 0: " << bad << '\n';
}
if (profile) std::cout << " hip network GPU: " << network->lastMilliseconds() * 1000.0 << " us\n";
#else
throw std::runtime_error("this build has no HIP support");
#endif
+6 -1
View File
@@ -569,7 +569,12 @@ void Engine::setFrame(const FrameImages& images, const InputParams& input, const
if ((input.flags & FrameFlagDisplayMotion) && !e.model.hasDisplayMotion)
throw std::runtime_error("display-resolution motion needs a model compiled by the current compile_k.py");
if (!(output.out2rnd[0] > 0 && output.out2rnd[0] <= 1 && output.out2rnd[1] > 0 && output.out2rnd[1] <= 1) ||
output.near2rnd[0] != output.out2rnd[0] || output.near2rnd[1] != output.out2rnd[1] ||
// Display-resolution motion bypasses the depth-neighbour lookup. NGX
// uses (1,1) for its unused nearest scale in that variant; standard
// frame construction uses the render ratio. Both are valid there.
(!(input.flags & FrameFlagDisplayMotion) &&
(output.near2rnd[0] != output.out2rnd[0] || output.near2rnd[1] != output.out2rnd[1])) ||
!(output.near2rnd[0] > 0 && output.near2rnd[0] <= 1 && output.near2rnd[1] > 0 && output.near2rnd[1] <= 1) ||
!std::isfinite(output.sharp) ||
!(std::isfinite(input.expo) && input.expo > 0 && std::isfinite(input.expoIn) && input.expoIn > 0 &&
std::isfinite(output.expo) && output.expo > 0 && std::isfinite(input.expoRatio) && input.expoRatio > 0 &&