bridge: let the launch-stats dump take a kernel filter
D4R_CUDA_LAUNCH_STATS=1 synchronizes after and dumps every argument of every launch, which costs about 1.2 GB and 5 s per frame - fine for a one-off inventory, too much for the edit/measure loop of a single kernel. D4R_CUDA_LAUNCH_STATS_FILTER=<substring> keeps only the kernels whose name contains it, and is checked before the synchronize so the rest of the frame is neither waited on nor written. Verified: with the filter set to swin_enc0 the dump is 54 files / 839 MB instead of 314 / 3.7 GB, the run takes 5 s instead of 17, and the 54 argument files are byte-identical to the ones the unfiltered run produced.
This commit is contained in:
1 parent
708b158332
commit
1a9762bb0e
2 files changed
+15
-2
No files matched your search
@@ -178,7 +178,7 @@ The release includes both fast and accuracy network sets. Full releases also inc
|
|||||||
```
|
```
|
||||||
|
|
||||||
It runs the prep and main kernels, saves every buffer after the first launch, and times 30 launches. The replay tool refuses captures with texture or surface objects; use the full harness for those kernels.
|
It runs the prep and main kernels, saves every buffer after the first launch, and times 30 launches. The replay tool refuses captures with texture or surface objects; use the full harness for those kernels.
|
||||||
3. **A per-layer oracle for kernels with texture and surface objects.** The replay capture above cannot describe those kernels (the replayers refuse them), but the bridge's other instrumentation can: with `D4R_CUDA_LAUNCH_STATS=1` and `D4R_CUDA_LAUNCH_DUMP_DIR=<dir>` every `cuLaunchKernel` is followed by a synchronize and a raw dump of every allocation, surface object and texture object named in its packed argument buffer, one file per launch and argument (`launch-NNN-<kernel>-argNNN-{buffer,surface,texture}-<shape>-fmtN.bin`). Run the harness once with the translated kernel and once with the native one and compare that kernel's files byte for byte: the surface files are the layer's own output, and a mismatch localises to that layer rather than to the frame. `cuda_dldn_engine_swin_enc0_kernel` dumps its weight image at arg040, its activation buffer at arg048 and four surfaces (1280x736x4 and 640x368x4 RGBA16F) as its outputs. The dump is large - 3 frames cost 3.7 GB - so keep the frame count small.
|
3. **A per-layer oracle for kernels with texture and surface objects.** The replay capture above cannot describe those kernels (the replayers refuse them), but the bridge's other instrumentation can: with `D4R_CUDA_LAUNCH_STATS=1` and `D4R_CUDA_LAUNCH_DUMP_DIR=<dir>` (with `D4R_CUDA_LAUNCH_STATS_FILTER=<substring>` to keep only the kernels being worked on, checked before the synchronize) every `cuLaunchKernel` is followed by a synchronize and a raw dump of every allocation, surface object and texture object named in its packed argument buffer, one file per launch and argument (`launch-NNN-<kernel>-argNNN-{buffer,surface,texture}-<shape>-fmtN.bin`). Run the harness once with the translated kernel and once with the native one and compare that kernel's files byte for byte: the surface files are the layer's own output, and a mismatch localises to that layer rather than to the frame. `cuda_dldn_engine_swin_enc0_kernel` dumps its weight image at arg040, its activation buffer at arg048 and four surfaces (1280x736x4 and 640x368x4 RGBA16F) as its outputs. The dump is large - 3 frames cost 3.7 GB - so keep the frame count small.
|
||||||
|
|
||||||
4. **Compare against a reference.** `kernels/tools/pwin_model.py` (K) and `swin_model.py` (M) are numpy models of the layers, written stage by stage against `ptxsim.py`, a small vectorised PTX interpreter that runs NVIDIA's kernels on the CPU. A bit-exact variant should match the model within f16 rounding; after any change, compare the new build's buffers with the previous build's.
|
4. **Compare against a reference.** `kernels/tools/pwin_model.py` (K) and `swin_model.py` (M) are numpy models of the layers, written stage by stage against `ptxsim.py`, a small vectorised PTX interpreter that runs NVIDIA's kernels on the CPU. A bit-exact variant should match the model within f16 rounding; after any change, compare the new build's buffers with the previous build's.
|
||||||
5. **Check the whole pipeline.** Replay captured frames through the harness with both kernel sets and compare the outputs with `kernels/tools/psnr.py`. `D4R_CUDA_KERNEL_PROFILE=1` makes the bridge time every kernel; `kernels/tools/kprof.py` summarises the log per frame.
|
5. **Check the whole pipeline.** Replay captured frames through the harness with both kernel sets and compare the outputs with `kernels/tools/psnr.py`. `D4R_CUDA_KERNEL_PROFILE=1` makes the bridge time every kernel; `kernels/tools/kprof.py` summarises the log per frame.
|
||||||
|
|||||||
@@ -1755,12 +1755,25 @@ static void summarize_launch(unsigned int sequence, CUfunction function_handle,
|
|||||||
if (extra == NULL || (uintptr_t)extra[0] != 1 || extra[1] == NULL ||
|
if (extra == NULL || (uintptr_t)extra[0] != 1 || extra[1] == NULL ||
|
||||||
(uintptr_t)extra[2] != 2 || extra[3] == NULL)
|
(uintptr_t)extra[2] != 2 || extra[3] == NULL)
|
||||||
return;
|
return;
|
||||||
|
/* D4R_CUDA_LAUNCH_STATS_FILTER restricts the summarise/dump to kernels whose
|
||||||
|
name contains it, before the synchronize, so iterating on one kernel does
|
||||||
|
not pay for, or write, the rest of the frame. */
|
||||||
|
static char stats_filter[128];
|
||||||
|
static int stats_filter_ready;
|
||||||
|
if (!stats_filter_ready)
|
||||||
|
{
|
||||||
|
const char* value = getenv("D4R_CUDA_LAUNCH_STATS_FILTER");
|
||||||
|
snprintf(stats_filter, sizeof(stats_filter), "%s", value != NULL ? value : "");
|
||||||
|
stats_filter_ready = 1;
|
||||||
|
}
|
||||||
|
const char* kernel = find_function_name(function_handle);
|
||||||
|
if (stats_filter[0] != '\0' && strstr(kernel, stats_filter) == NULL)
|
||||||
|
return;
|
||||||
CUCTX_SYNCHRONIZE_FN synchronize = (CUCTX_SYNCHRONIZE_FN)find_zluda_symbol("cuCtxSynchronize");
|
CUCTX_SYNCHRONIZE_FN synchronize = (CUCTX_SYNCHRONIZE_FN)find_zluda_symbol("cuCtxSynchronize");
|
||||||
CUMEMCPYDTOH_FN copy_to_host = (CUMEMCPYDTOH_FN)find_zluda_symbol("cuMemcpyDtoH_v2");
|
CUMEMCPYDTOH_FN copy_to_host = (CUMEMCPYDTOH_FN)find_zluda_symbol("cuMemcpyDtoH_v2");
|
||||||
if (synchronize == NULL || copy_to_host == NULL)
|
if (synchronize == NULL || copy_to_host == NULL)
|
||||||
return;
|
return;
|
||||||
const CUresult sync_result = synchronize();
|
const CUresult sync_result = synchronize();
|
||||||
const char* kernel = find_function_name(function_handle);
|
|
||||||
tracef("launch[%u] %s synchronize result=%d", sequence, kernel, sync_result);
|
tracef("launch[%u] %s synchronize result=%d", sequence, kernel, sync_result);
|
||||||
const unsigned char* arguments = (const unsigned char*)extra[1];
|
const unsigned char* arguments = (const unsigned char*)extra[1];
|
||||||
const size_t argument_bytes = *(const size_t*)extra[3];
|
const size_t argument_bytes = *(const size_t*)extra[3];
|
||||||
|
|||||||
Reference in new issue
Block a user