kernels/rr: staged-image Swin layer engine prototypes
Per-layer staged-image engine for the eleven translated denoiser Swin layers: descriptors (verified offsets per phase), cut recipes that splice an isolated native phase into NVIDIA's PTX, wave templates, epilogue notes and GPU probes. The isolated phase-8 splice produces a byte-identical final 4K frame but is slower than the translated body (14.90 vs 6.59 ms) - a correctness experiment, not a shipping improvement; docs/rr-engine.md records the design and the next direct-converter win (B operands built from transposed accumulators).
This commit is contained in:
woutermans committed
1 parent
798e25611c
commit
0fe3a28cb7
58 files changed
+28330
-5
No files matched your search
@@ -0,0 +1,845 @@
|
||||
# `cuda_dldn_engine_swin_enc3_kernel` — the epilogues of phases 1…48
|
||||
|
||||
Companion to `kernels/rr/rr_layer_spec.py enc3`. Everything below was read out of
|
||||
`~/.cache/d4r-rr-corpus/0016-PREPASS_ENTRYPOINT_NAME.ptx`, entry
|
||||
`cuda_dldn_engine_swin_enc3_kernel`.
|
||||
|
||||
## 0. Conventions
|
||||
|
||||
* **`sNNNN` is a statement index** in the linearisation `rr_layer_spec.py`
|
||||
documents (statements of the entry body; a wrapped `{ … }` block and a 4-line
|
||||
`mma.sync` count as one statement, numbered at its first line). **`LNNNNN` is
|
||||
the physical line** of the same instruction in the corpus file, so every claim
|
||||
can be checked with `sed -n 'LNNNNN,LNNNNN p' …ptx`.
|
||||
* An **epilogue** is the run of statements between one phase's last `mma` and the
|
||||
next phase's first `mma`. The phase table gives the ranges; this file names
|
||||
what each one *computes*.
|
||||
* `W` denotes the prepared weight image: `ld.param.u64 %rd2,
|
||||
[cuda_dldn_engine_swin_enc3_kernel_param_0+40]` (s8). Every
|
||||
`ld.weak.global.ca.v4.u32` in an epilogue reads `W + 16*laneid + imm`, and the
|
||||
global alias `%rd74 = cvta.to.global.u64 %rd2` (s5564) is what the table loads
|
||||
in the epilogues use.
|
||||
* `S` denotes the plane arena: `param_0+48`, reached as
|
||||
`ld.param.u64 %rd556, [%rd1+-32]` with `%rd1 = param_0+80` (`add.s64 %rd1,
|
||||
%rd7, 80`, s4) then `cvta.to.global.u64 %rd4, %rd556` (s16148–s16149).
|
||||
* The C/D register layout and the weight-addressing formula are fixed (see the
|
||||
settled notes): `C/D reg0 = (row g, cols 2t,2t+1)`, `reg1 = (row g+8, same cols)`,
|
||||
`g = laneid>>2`, `t = laneid&3`.
|
||||
* `rr_layer_spec.py`'s printed `M`/`N`/`K` come from `geom()`, i.e. from the
|
||||
distinct A-fragment count (`M = 16·nA`), the distinct B-fragment count
|
||||
(`N = 8·nB`) and the accumulator-chain length (`K = 32·len`). The phase
|
||||
object's own `w_min..w_max`/`n_tiles`/`k_slots` fields use a *different*
|
||||
(512-byte-block) convention and disagree with `geom()` on the stem phases; §2
|
||||
uses `geom()` and checks the arithmetic explicitly.
|
||||
* Anything I could not pin down from the PTX is in §6 under **uncharacterised**.
|
||||
|
||||
## 1. Shared vocabulary
|
||||
|
||||
### 1.1 Addresses and parameters an epilogue touches
|
||||
|
||||
| symbol | how it is obtained | statements |
|
||||
|---|---|---|
|
||||
| `W` (weight image, 315732 bytes) | `ld.param.u64 %rd2, [param_0+40]` | s8 |
|
||||
| `W`(global alias) | `cvta.to.global.u64 %rd74, %rd2` | s5564 |
|
||||
| `S` (plane arena, `param+48`) | `ld.param.u64 %rd556, [%rd1+-32]` → `cvta.to.global.u64 %rd4, %rd556` | s16148–s16149 |
|
||||
| output buffer (`param+56`) | `ld.param.u64 %rd5, [%rd682+56]` | s16581 |
|
||||
| input plane (`param+8`) | `ld.param.u64 %rd9, [param_0+8]` → `cvta.to.global.u64 %rd3, %rd9` | s16–s17 |
|
||||
| extents `(ex, ey)` | `ld.param.v2.u32 {%r407, %r408}, [param_0+0]` | s13 |
|
||||
| block origin `(x0, y0)` | `%r402 = ctaid.x<<3`, `%r1 = %r402 - %rs166`; `%r406 = ctaid.y<<3`, `%r2 = %r406 - %rs167`; `%rs166/%rs167` from `ld.param.v2.u16 [param_0+80]` | s2, s3, s7, s10, s11, s12, s5 |
|
||||
| `%rd7` = `param_0` | `mov.b64 %rd7, cuda_dldn_engine_swin_enc3_kernel_param_0` | s1 |
|
||||
| shared arena base `%r4244` | `mov.u32 %r4244, _ZZ33cuda_dldn_engine_swin_enc3_kernel33DldnEngineSwinEncParamsStructBaseE4smem` | s6002 |
|
||||
| `%r235` = `%r4244 + (tid.z<<11)` | `shl.b32 %r4243, %r12, 11` then `add.s32 %r235, %r4244, %r4243` | s6001, s6003 |
|
||||
| per-lane weight address | `mul.wide.u32 %rd, %laneid, 16` + `add.s64 %rd, %rd2, %rd` | e.g. s6115–s6117 |
|
||||
| per-lane *table* address | `shl.b32 %r, %laneid, 2` → `cvt.u64.u32` → `and.b64 %rd, %r, 12` → `add.s64 %rd, %rd74, %rd` = `W + ((laneid<<2)&12)` | e.g. s10558–s10561 |
|
||||
|
||||
The last line matters exactly as in enc0: a `ld.global.v2.u16` / `ld.global.u32`
|
||||
with that addressing reads **the same 16-byte record from the four lanes with
|
||||
equal `laneid&3`**, so those tables are indexed by column, not by row.
|
||||
|
||||
The shadow copy `ld.param.u64 %rd947, [param_0+40]` appears again at s20229 in the
|
||||
final epilogue — the weight image pointer is re-read, not cached.
|
||||
|
||||
### 1.2 Constants that survive between epilogues
|
||||
|
||||
| register | first definition | literal | meaning |
|
||||
|---|---|---|---|
|
||||
| `%f476` | s6585 `mov.f32 %f476, 0f3C8CCB50` | 0.017186790704727173 | f16 `0x2466` = 0.017181396484375 (softmax input scale) |
|
||||
| `%fd383` | s6589 `mov.f64 %fd383, 0dBFE1CC0000000000` | −0.55615234375 | f16 clamp lower bound |
|
||||
| `%fd385` | s6594 `mov.f64 %fd385, 0d3FE1CC0000000000` | +0.55615234375 | f16 clamp upper bound |
|
||||
| `%f478` | s6600 `mov.f32 %f478, 0f3F6D6000` | 0.92724609375 | cubic coefficient |
|
||||
| `%f480` | s6605 `mov.f32 %f480, 0f3FB00000` | 1.375 | cubic constant term |
|
||||
| `%fd1` | s4856 `mov.f64 %fd1, 0d3F20000000000000` | 2⁻¹³ | norm epsilon |
|
||||
| `%f989` | s11656 `mov.f32 %f989, 0f3ED306EB` | 0.4121621549129486 | activation coefficient |
|
||||
| `%f990` | s11659 `mov.f32 %f990, 0f3DA60DD6` | 0.0810810774564743 | activation coefficient |
|
||||
| `%f991` | s11662 `mov.f32 %f991, 0f3F000000` | 0.5 | activation offset |
|
||||
| `%f992` | s11665 `mov.f32 %f992, 0f40000000` | 2.0 | activation clamp |
|
||||
|
||||
`%f989…%f992` are defined inside phase 15's epilogue and re-materialised (as
|
||||
`mov.f32` + `cvt.rn.f16.f32` pairs) inside the epilogue of every odd phase
|
||||
17…45, e.g. s12054–s12064 (E17) — unlike enc0 the *f32* constants are not reused,
|
||||
only their values.
|
||||
|
||||
Live across the whole mma region (defined in the prologue, read by every warp
|
||||
reduction):
|
||||
|
||||
| register | statement | value | used as |
|
||||
|---|---|---|---|
|
||||
| `%r4016` | s4416 | 1 | `shfl.sync.bfly` `b` operand, bit 0 |
|
||||
| `%r4025` | s4805 | 2 | `shfl.sync.bfly` `b` operand, bit 1 |
|
||||
| `%r4027` | s4798 | −1 | `shfl.sync.bfly` membermask |
|
||||
|
||||
The `c` (clamp/segment) operand of every butterfly is built as
|
||||
`%r = (WARP_SZ << 8) - 8192` then `or.b32 %r, %r, 31` (e.g. s8785–s8787), i.e.
|
||||
31 for a 32-lane warp.
|
||||
|
||||
The long-lived **residual vector**: 48 `mov.b32 {%rsA, %rsB}, %r3713+4k`
|
||||
statements at s4480 ff. split `%r3713, %r3717, %r3721 …` into f16 halves
|
||||
(`%rs1460/%rs1463`, `%rs1466/%rs1469`, `%rs1472/%rs1475`, …). Each `%r3713+4k` is
|
||||
`cvt.rn.f16x2.e4m3x2 %r3713, %rs168` (s4415), i.e. the dequantisation of two
|
||||
packed e4m3 bytes loaded from the input plane by `ld.global.v2.u16 {%rs168,
|
||||
%rs169}, [%rd11]` with `%rd11 = %rd3 + …` and `%rd3 = cvta.to.global.param_0+8`
|
||||
(s173–s175). These halves are added to the column tables in E10 (§3/E10) and
|
||||
read nowhere else.
|
||||
|
||||
### 1.3 Quantisation helpers
|
||||
|
||||
* `cvt.rn.satfinite.e4m3x2.f16x2` — the requantisation used everywhere
|
||||
(`f16x2_to_e4m3x2_satfinite_bits` in `ptx/lib/zluda_ptx_impl.cpp`).
|
||||
* `cvt.rn.f16x2.e4m3x2` — its inverse, used only in the prologue (s4415 ff.).
|
||||
* `rcp.approx.ftz.f32`, `rsqrt.approx.ftz.f32`, `ex2.approx.ftz.f32`,
|
||||
`div.approx.ftz.f32` — always wrapped as
|
||||
`cvt.f32.f16 → op → cvt.rn.f16.f32` (each step rounded through f16).
|
||||
|
||||
### 1.4 The shared arena
|
||||
|
||||
`_ZZ…E4smem` is declared 12800 bytes. The mma A operands of phases 1…8 come from
|
||||
24 `ld.shared.v4.u32` at `%r4244 + 16*laneid + 512*k`, `k = 0…23`
|
||||
(s6090–s6113, lines 9771–9794); phases 11…14 come from 4 such loads at
|
||||
`%r4244 + (tid.z<<9) + 16*laneid + {0, 2048, 4096, 6144}` (s10537, s10693, s10732,
|
||||
s10770); phase 47 comes from 64 `ld.shared.u32` (s16502–s16574). The arena is
|
||||
written by the prologue (s6054–s6083), by the P*V epilogue E10 (s10491–s10530)
|
||||
and by E46 (s16348–s16480). See §6/U3.
|
||||
|
||||
## 2. Phase table summary
|
||||
|
||||
Reproduced from `rr_layer_spec.py enc3` with `M`, `N`, `K` from `geom()` (fragment
|
||||
counts and chain lengths), and `check = (M/16)·(N/8)·(K/32)`:
|
||||
|
||||
| # | stmt range | lines | M | N | K | nA | nB | chains | klens | mma | check | =? | weight bytes | C bias bytes | epi |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | s6124–s6147 | L9812–9973 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 256..780 | – | 7 |
|
||||
| 2 | s6155–s6178 | L9993–10154 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 4352..4876 | – | 7 |
|
||||
| 3 | s6186–s6209 | L10174–10335 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 1280..1804 | – | 7 |
|
||||
| 4 | s6217–s6240 | L10355–10516 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 5376..5900 | – | 7 |
|
||||
| 5 | s6248–s6271 | L10536–10697 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 2304..2828 | – | 7 |
|
||||
| 6 | s6279–s6302 | L10717–10878 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 6400..6924 | – | 7 |
|
||||
| 7 | s6310–s6333 | L10898–11059 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 3328..3852 | – | 7 |
|
||||
| 8 | s6341–s6364 | L11079–11240 | 96 | 32 | 32 | 6 | 4 | 24 | 1 | 24 | 24 | Y | 7424..7948 | – | 171 |
|
||||
| 9 | s6536–s6583 | L11628–11957 | 64 | 96 | 32 | 4 | 12 | 48 | 1 | 48 | 48 | Y | – | 8448..20236 | 3843 |
|
||||
| 10 | s10427–s10474 | L22107–22436 | 192 | 96 | 96 | 12 | 12 | 16 | 3 | 48 | 432 | **N** | – | – | 199 |
|
||||
| 11 | s10674–s10689 | L22922–23027 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 20736..24332 | – | 23 |
|
||||
| 12 | s10713–s10728 | L23077–23182 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 45312..48908 | – | 23 |
|
||||
| 13 | s10752–s10767 | L23232–23337 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 69888..73484 | – | 23 |
|
||||
| 14 | s10791–s10806 | L23387–23492 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 94464..98060 | – | 832 |
|
||||
| 15 | s11639–s11654 | L25484–25589 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 99072..102668 | 103168..103216 | 289 |
|
||||
| 16 | s11944–s11959 | L26459–26564 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 103232..106828 | – | 76 |
|
||||
| 17 | s12036–s12051 | L26731–26836 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 107584..111180 | 111680..111728 | 168 |
|
||||
| 18 | s12220–s12235 | L27391–27496 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 111744..115340 | – | 76 |
|
||||
| 19 | s12312–s12327 | L27663–27768 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 115840..119436 | 119936..119984 | 168 |
|
||||
| 20 | s12496–s12511 | L28323–28428 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 120000..123596 | – | 76 |
|
||||
| 21 | s12588–s12603 | L28595–28700 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 124096..127692 | 128192..128240 | 168 |
|
||||
| 22 | s12772–s12787 | L29255–29360 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 128256..131852 | – | 76 |
|
||||
| 23 | s12864–s12879 | L29527–29632 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 132352..135948 | 136448..136496 | 168 |
|
||||
| 24 | s13048–s13063 | L30187–30292 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 136512..140108 | – | 76 |
|
||||
| 25 | s13140–s13155 | L30459–30564 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 140608..144204 | 144704..144752 | 168 |
|
||||
| 26 | s13324–s13339 | L31119–31224 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 144768..148364 | – | 76 |
|
||||
| 27 | s13416–s13431 | L31391–31496 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 148864..152460 | 152960..153008 | 168 |
|
||||
| 28 | s13600–s13615 | L32051–32156 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 153024..156620 | – | 76 |
|
||||
| 29 | s13692–s13707 | L32323–32428 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 157120..160716 | 161216..161264 | 168 |
|
||||
| 30 | s13876–s13891 | L32983–33088 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 161280..164876 | – | 76 |
|
||||
| 31 | s13968–s13983 | L33255–33360 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 165376..168972 | 169472..169520 | 168 |
|
||||
| 32 | s14152–s14167 | L33915–34020 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 169536..173132 | – | 76 |
|
||||
| 33 | s14244–s14259 | L34187–34292 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 173632..177228 | 177728..177776 | 168 |
|
||||
| 34 | s14428–s14443 | L34847–34952 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 177792..181388 | – | 76 |
|
||||
| 35 | s14520–s14535 | L35119–35224 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 181888..185484 | 185984..186032 | 168 |
|
||||
| 36 | s14704–s14719 | L35779–35884 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 186048..189644 | – | 76 |
|
||||
| 37 | s14796–s14811 | L36051–36156 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 190144..193740 | 194240..194288 | 168 |
|
||||
| 38 | s14980–s14995 | L36711–36816 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 194304..197900 | – | 76 |
|
||||
| 39 | s15072–s15087 | L36983–37088 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 198400..201996 | 202496..202544 | 168 |
|
||||
| 40 | s15256–s15271 | L37643–37748 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 202560..206156 | – | 76 |
|
||||
| 41 | s15348–s15363 | L37915–38020 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 206656..210252 | 210752..210800 | 168 |
|
||||
| 42 | s15532–s15547 | L38575–38680 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 210816..214412 | – | 76 |
|
||||
| 43 | s15624–s15639 | L38847–38952 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 214912..218508 | 219008..219056 | 168 |
|
||||
| 44 | s15808–s15823 | L39507–39612 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 219072..222668 | – | 76 |
|
||||
| 45 | s15900–s15915 | L39779–39884 | 64 | 128 | 128 | 4 | 16 | 4 | 4 | 16 | 256 | **N** | 223168..226764 | 227264..227312 | 168 |
|
||||
| 46 | s16084–s16099 | L40439–40544 | 16 | 128 | 32 | 1 | 16 | 16 | 1 | 16 | 16 | Y | 227328..230924 | – | 564 |
|
||||
| 47 | s16664–s16727 | L41319–41760 | 256 | 512 | 512 | 16 | 64 | 4 | 16 | 64 | 16384 | **N** | 231424..247308 | 313344..313392 | 125 |
|
||||
| 48 | s16853–s16860 | L41931–41980 | 64 | 64 | 128 | 4 | 8 | 2 | 4 | 8 | 128 | **N** | 313664..315212 | 315712..315728 | 323 |
|
||||
|
||||
**Arithmetic check.** Rows 1–9, 11–16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36,
|
||||
38, 40, 42, 44 and 46 satisfy
|
||||
`M/16 · N/8 · K/32 == mma`, i.e. every A×B tile of the M×N tile grid is
|
||||
evaluated exactly once, so `M`/`N`/`K` are the true GEMM shape.
|
||||
|
||||
Rows 10, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47 and 48
|
||||
do **not**: the phase is *banded* (a diagonal block of the M×N grid) and the
|
||||
printed `M`, `N` are the *extents spanned by the fragment operands*, not the shape
|
||||
of the result. The chain structure makes the real shape explicit:
|
||||
|
||||
* **P10** (`48 mma`, 16 chains of 3): chain `c` is
|
||||
`(A_{3·(c>>2)+k}, B_{2·(c&3)+k})` for `k = 0,1,2` — 4 row tiles × 4 column-pair
|
||||
groups, each accumulating 3 k-steps. Real shape **64×32×96**, all
|
||||
4×4 = 16 output tiles evaluated. (Same shape as enc0's P6.)
|
||||
* **P15/17/…/45** (`16 mma`, 4 chains of 4): chain 0 is
|
||||
`(A0,B0) (A1,B2) (A2,B4) (A3,B6)`, chain 1 is `(A0,B1) (A1,B3) (A2,B5) (A3,B7)`,
|
||||
chain 2 repeats with `B8…B15`, chain 3 with `B9…B15` — 4 row tiles × 2 column
|
||||
tiles per 8-B group, 4 k-steps. Real shape **64×64×128** with only the
|
||||
4×2 = 8 diagonal output tiles evaluated (of a 4×8 grid).
|
||||
* **P47** (`64 mma`, 4 chains of 16): chain 0 is
|
||||
`(A_j, B_{2j})`, chain 1 `(A_j, B_{2j+1})`, chains 2/3 `B_{32+2j}`, `B_{32+2j+1}`,
|
||||
`j = 0…15`. 16 k-steps. Real shape **256×256×512** over 4 output tiles.
|
||||
* **P48** (`8 mma`, 2 chains of 4): `(A_j, B_{2j})` / `(A_j, B_{2j+1})`,
|
||||
`j = 0…3`. Real shape **64×32×128** over 2 tiles.
|
||||
* **P15…P45 odd** likewise.
|
||||
|
||||
**Phases that read no weight-image B operand:** **P9** (`Bo = [None, None]` on all
|
||||
48 mma — its `B` is a register operand packed from P7's D, §3/E8) and **P10**
|
||||
(its `B` is packed from the `movmatrix` of P8's D, §3/E9(f)). Every other phase's
|
||||
`B` is loaded from `W`.
|
||||
|
||||
**Weight-image extent.** The largest byte offset used anywhere is 315728 (P48's
|
||||
second C seed, `ld.global.u32 [%rd653+315728]`, s16852; the first is `+315712`,
|
||||
s16851). With the 4-byte read that makes the image **315732 bytes**; 1654
|
||||
distinct weight-image byte offsets are touched (`weight_image_size`). The single largest interior gap is
|
||||
`247308…313344` — 66036 bytes between the end of P47's B block and P47's bias that
|
||||
no phase in this entry reads (the analogous hole in enc4 is 82420 bytes, §5/U9 of
|
||||
`rrswin_enc4_epilogues.md`).
|
||||
|
||||
## 3. The epilogues
|
||||
|
||||
Phases 1…8 (the stem), 11…45 (the norm + MLP block) and 46…48 (the merge) are
|
||||
highly repetitive; each distinct body is written out once and the repeats are
|
||||
tabulated.
|
||||
|
||||
### E1…E7 — s6148–s6154, s6179–s6185, s6210–s6216, s6241–s6247, s6272–s6278, s6303–s6309, s6334–s6340: next-B loads
|
||||
|
||||
Seven identical 7-statement epilogues (`add.s64`×3, 2 `ld.weak.global.ca.v4.u32`,
|
||||
`mov.u32`, `mul.wide.u32`). Each issues the two `v4.u32` fragments of the *next*
|
||||
phase's B operand at `W + 16*laneid + {base, base+512}`:
|
||||
|
||||
| epilogue | range | loads (statement / line) | offsets | next phase's W block |
|
||||
|---|---|---|---|---|
|
||||
| E1 | s6148–s6154 | s6152 / L9986, s6154 / L9990 | 4352, 4864 | P2 `4352..4876` |
|
||||
| E2 | s6179–s6185 | s6183 / L10167, s6185 / L10171 | 1280, 1792 | P3 `1280..1804` |
|
||||
| E3 | s6210–s6216 | s6214 / L10348, s6216 / L10352 | 5376, 5888 | P4 `5376..5900` |
|
||||
| E4 | s6241–s6247 | s6245 / L10529, s6247 / L10533 | 2304, 2816 | P5 `2304..2828` |
|
||||
| E5 | s6272–s6278 | s6276 / L10710, s6278 / L10714 | 6400, 6912 | P6 `6400..6924` |
|
||||
| E6 | s6303–s6309 | s6307 / L10891, s6309 / L10895 | 3328, 3840 | P7 `3328..3852` |
|
||||
| E7 | s6334–s6340 | s6338 / L11072, s6340 / L11076 | 7424, 7936 | P8 `7424..7948` |
|
||||
|
||||
Nothing else. **Outputs:** the 8 registers of the next phase's B.
|
||||
|
||||
### E8 — s6365–s6535: the score-phase bias tiles and the phase-9 operands
|
||||
|
||||
171 statements: 24 `ld.weak.global.ca.v4.u32`, 80
|
||||
`cvt.rn.satfinite.e4m3x2.f16x2`, 40 `mov.b32`, 25 `add.s64`, plus one
|
||||
`mov.u32`/`mul.wide.u32` per-lane address pair.
|
||||
|
||||
```
|
||||
s6368 L11249: add.s64 %rd94, %rd425, 8448
|
||||
s6369 L11253: ld.weak.global.ca.v4.u32 { %r6280,%r6281,%r6282,%r6283},[%rd94]
|
||||
s6415 L11345: ld.weak.global.ca.v4.u32 { %r6372,…%r6375},[%rd117] // 8448+23*512 = +20224
|
||||
s6416 L11348: cvt.rn.satfinite.e4m3x2.f16x2 %rs1942, %r5810
|
||||
s6489 L11522: cvt.rn.satfinite.e4m3x2.f16x2 %rs1989, %r5790
|
||||
s6535 L11626: mov.b32 %r6933, {%rs2034, %rs2035}
|
||||
```
|
||||
|
||||
1. **Score-bias C seeds** (s6368–s6415): 24 `v4.u32` loads at
|
||||
`W + 16*laneid + 8448 + 512*i`, `i = 0…23` → `%r6280 … %r6375`
|
||||
(96 registers = the 96 `C` operands of phase 9). The index is the same
|
||||
`4*(n>>1) + m` scheme enc0 documents: `4*(n>>1)+m` runs 0…23 for the 4 m-tiles
|
||||
and 6 `(n>>1)` groups, and phase 9's own mma confirm the assignment
|
||||
(`s6536 C=%r6280,%r6281` first, `s6538 C=%r6296,%r6297` = the `+2048` tile).
|
||||
Byte block `[8448, 20236]` = 11792 bytes of f16, structurally identical to
|
||||
enc0's `[8384, 20672)`.
|
||||
2. **B packs** (s6418–s6487, sources from s6416): the first 24 `mov.b32`
|
||||
destinations
|
||||
(`%r6822 %r6823 %r6832 %r6833 … %r6932 %r6933`) = phase 9's twelve B fragments
|
||||
(24 b32). Every source pair is
|
||||
`cvt.rn.satfinite.e4m3x2.f16x2` of two registers written by **phase 7's mma**
|
||||
(`%r5790 … %r6021`), verified by walking each `mov.b32`'s operands through its
|
||||
two `cvt` statements (e.g. `s6417: cvt %rs1941, %r5790`, `s6416: cvt %rs1942,
|
||||
%r5810`, `s6418: mov.b32 %r6822, {%rs1941, %rs1942}`).
|
||||
3. **A packs** (s6490–s6535): the remaining 16 `mov.b32` destinations
|
||||
(`%r6568 %r6569 %r6570 %r6571`, `%r6688…%r6691`, `%r6808…%r6811`,
|
||||
`%r6928…%r6931`) = phase 9's four A fragments (16 b32), also from phase 7's D
|
||||
(`s6489: cvt %rs1989, %r5790`).
|
||||
|
||||
**Phase 9's A fragment 0 and B fragment 0 are bit-identical**: both are
|
||||
`{pack(%r5790,%r5810), …}` — `%r6568` (s6489–s6491) and `%r6822` (s6416–s6418)
|
||||
read the same two source registers. Whatever the score GEMM's two operands mean,
|
||||
fragment 0 of each is the same value, exactly as in enc0 (E9).
|
||||
|
||||
**Inputs:** phase 7's 48 D registers, phase 8's `%r10141`-adjacent state.
|
||||
**Outputs:** phase 9's 16 A + 24 B registers and its 96 C seeds.
|
||||
|
||||
### E9 — s6584–s10426: the 64×96 score softmax
|
||||
|
||||
3843 statements, **zero memory access**. This is the enc0 E5 block verbatim with
|
||||
shifted registers; the op histogram is the same (1048 `mov.b32`, 672
|
||||
`cvt.rn.f16.f32`, 384 `cvt.rn.f16.f64`, 192 each `fma.rn.f16x2` /
|
||||
`cvt.f32.f16` / `rcp.approx.ftz.f32` / `mul.f16`, 96 each `max.f16x2`/`min.f16x2`/
|
||||
`neg.f16x2`/`mul.f16x2`/`and.b32`, 48 `movmatrix`, 16 `shfl.sync.bfly`, 144 packs).
|
||||
|
||||
Sub-ranges (first/last statement with the physical line):
|
||||
|
||||
| range | statements | what |
|
||||
|---|---|---|
|
||||
| s6584–s8701 | 2118 | 96 per-element exponent groups (scale, clamp, cubic, mantissa shift/and) |
|
||||
| s8702–s8781 | 80 | in-lane partial sum trees (`add.f16x2`) |
|
||||
| s8782–s8916 | 135 | 16 `shfl.sync.bfly.b32` (s8788/L18834 … s8909/L19138) → 8 row sums |
|
||||
| s8918–s9679 | 769 | 192 `rcp.approx.ftz.f32` → 96 reciprocals |
|
||||
| s9685–s10161 | 477 | 192 `mul.f16` → 96 probabilities |
|
||||
| s10163–s10210 | 48 | `movmatrix.sync.trans.aligned.m8n8.b16` of **phase 8's** 48 D registers |
|
||||
| s10211–s10425 | 216 | 144 `cvt.rn.satfinite.e4m3x2.f16x2` packs |
|
||||
|
||||
**(a) per-element exponent, s6584–s8701.** Per f16 half `x` of each of phase 9's
|
||||
96 D f16x2 registers:
|
||||
|
||||
```
|
||||
m = clamp(f16(x * 0.017181396484375), -0.55615234375, +0.55615234375)
|
||||
s6588 mul.f16x2, s6593 max.f16x2, s6598 min.f16x2
|
||||
u = f16(1.375 + m*(0.92724609375 - m^2))
|
||||
s6604 fma.rn.f16x2 (m*(-m) + 0.92724609375), s6609 fma.rn.f16x2 (m*u + 1.375)
|
||||
E = (u_bits << 5) & 0x7FE07FE0 // s6610 shl.b32, s6611 and.b32
|
||||
```
|
||||
|
||||
i.e. the same "cubic in the exponent field" approximation enc0's E5(a) derives and
|
||||
which the acceptance text asks about. The constants are re-materialised per group
|
||||
(`s6585 mov.f32 %f476 …`, `s6589/s6594 mov.f64 %fd383/%fd385 …`) — unlike enc0,
|
||||
this layer's E9 does not hoist them.
|
||||
|
||||
**(b) row sums, s8702–s8781 + s8782–s8916.** 96 `expval` registers are added in
|
||||
eight groups of six (`add.f16x2`), then across lane bits 0 and 1 with
|
||||
`shfl.sync.bfly` using `%r4016 = 1` and `%r4025 = 2`, then the f16x2 halves are
|
||||
folded with `add.f16` and broadcast (`s8796 mov.b32 {%rs2790,%rs2791}, %r9116` /
|
||||
`s8797 add.f16` / `s8798 mov.b32 %r9313, {%rs2789,%rs2789}`).
|
||||
|
||||
**There is no row maximum anywhere in this epilogue.** `max.f16x2`/`min.f16x2`
|
||||
(96 each, s6593…s8691) are the clamp of (a); the only shuffles are the 16
|
||||
`shfl.sync.bfly` above. (Contrast E48(b), which *does* take a row max.)
|
||||
|
||||
**(c) reciprocal / probability, s8918–s10161.** 192 `rcp.approx.ftz.f32` wrapped
|
||||
as `cvt.f32.f16 → rcp → cvt.rn.f16.f32` on the 8 row sums (each inverted once per
|
||||
row-tile), then 192 `mul.f16` giving `p = expval * (1/rowsum)`.
|
||||
|
||||
**(d) V transpose, s10163–s10210.** Exactly 48
|
||||
`movmatrix.sync.trans.aligned.m8n8.b16` on phase **8**'s 48 D registers
|
||||
(`%r6039 %r6040 %r6049 %r6050 … %r6270`; `s10163 L21459: movmatrix… %r9462,
|
||||
%r6039`, `s10210 L21600: … %r9556, %r6270`) → 48 b32 of transposed V.
|
||||
|
||||
**(e) packs, s10211–s10425.** 144 packs: s10211–s10314 from the movmatrix
|
||||
results → phase 10's 12 B fragments (`%r10068 …%r10179`, e.g. `s10213: mov.b32
|
||||
%r10068, {%rs3389, %rs3390}`); the rest from the probabilities → phase 10's 12 A
|
||||
fragments (`%r9774 …%r10177`, e.g. `s10427 A={%r9774,%r9775,%r9776,%r9777}`).
|
||||
|
||||
**Inputs:** phase 9's 96 D, phase 8's 48 D, `%r4016/%r4025/%r4027`.
|
||||
**Outputs:** phase 10's 48 A / 24 B registers (plus the `%f476`, `%fd383`,
|
||||
`%fd385`, `%f478`, `%f480` constants). **Memory:** none.
|
||||
|
||||
### E10 — s10475–s10673: P·V epilogue — pack, shared staging, residual table, next B/C
|
||||
|
||||
199 statements. Histogram: `add.f16`×64, `mov.b32`×48,
|
||||
`cvt.rn.satfinite.e4m3x2.f16x2`×32, `ld.global.v2.u16`×16, `add.s64`×10,
|
||||
`ld.weak.global.ca.v4.u32`×8, `mov.u32`×4, `shl.b32`×4, `st.shared.v4.u32`×4,
|
||||
`add.s32`×3, `bar.sync`×2, `ld.shared.v4.u32`×1, `mul.wide.u32`×1,
|
||||
`cvt.u64.u32`×1, `and.b64`×1.
|
||||
|
||||
**(a) require, s10479–s10530.** 32 packs of phase 10's D → 16 b32
|
||||
(`%r22010 … %r22025`, `s10487 L22472: mov.b32 %r22010, {%rs3539, %rs3540}`),
|
||||
stored one `v4.u32` per 512 bytes at
|
||||
`%r22009 = %r235 + (laneid<<4)` (`s10478 add.s32 %r22009, %r235, %r22008`,
|
||||
`%r22008 = laneid<<4` s10477):
|
||||
`s10491 st.shared.v4.u32 [%r22009], …`, `s10504 [+512]`, `s10517 [+1024]`,
|
||||
`s10530 [+1536]`.
|
||||
|
||||
**(b) stage read-back, s10531–s10541.** `s10531 bar.sync 0`, then one
|
||||
`ld.shared.v4.u32 [%r22029]` at
|
||||
`%r22029 = %r4244 + (tid.z<<9) + (laneid<<4)` (`s10532–s10536`) → `%r22030 …
|
||||
%r22033` (one 16-byte A fragment). These are **phase 11's A operand** (P11
|
||||
`A = %r22030,%r22031,%r22032,%r22033`).
|
||||
|
||||
**(c) next B, s10541–s10556.** 8 `v4.u32` at `W + 16*laneid + 20736 + 512*j`,
|
||||
`j = 0..7` → phase 11's 16 B fragments (block `[20736, 24332)`).
|
||||
|
||||
**(d) residual column table, s10557–s10577.** 16 `ld.global.v2.u16` at
|
||||
`W + ((laneid<<2)&12) + {98560 + 16*i}`, `i = 0…15` → 32 f16
|
||||
(`%rs5396 … %rs5427`) covering the 128 columns of phase 11's output.
|
||||
|
||||
**(e) residual add, s10578–s10673.** 64 `add.f16` — two per f16x2 — each
|
||||
`table[column] + <half of the long-lived prologue residual>`, e.g.
|
||||
|
||||
```
|
||||
s10579 L22638: add.f16 %rs3565,%rs5426,%rs1460 // %rs5426 = W+98560, %rs1460 = residual
|
||||
s10578 L22634: add.f16 %rs3568,%rs5427,%rs1463
|
||||
s10580 L22641: mov.b32 %r10258, {%rs3565, %rs3568}
|
||||
```
|
||||
|
||||
The 32 results `%r10258 %r10259 %r10268 … %r10409` are **phase 11's C operands**
|
||||
(`s10674 C={%r10258,%r10259}`). So P11 computes
|
||||
`D11 = A11·B11 + (table + residual)`.
|
||||
|
||||
**Inputs:** phase 10's D, phase 8's D staging, `%r4244/%r235`, `W`.
|
||||
**Outputs:** phase 11's A (4 regs), B (32 regs) and C (32 regs).
|
||||
|
||||
### E11…E13 — s10690–s10712, s10729–s10751, s10768–s10790: next-A from shared, next-B from W
|
||||
|
||||
Three identical 23-statement epilogues (`add.s64`×9, 8 `v4.u32`, `mov.u32`×2,
|
||||
`shl.b32`, `add.s32`, 1 `ld.shared.v4.u32`, `mul.wide.u32`):
|
||||
|
||||
```
|
||||
s10692 L23037: add.s32 %r22036, %r22027, %r22035 // %r22027 = %r4244 + (tid.z<<9)
|
||||
s10693 L23038: ld.shared.v4.u32 {…}, [%r22036+2048]
|
||||
s10697 L23044: add.s64 %rd126, %rd433, 45312
|
||||
s10698 L23046: ld.weak.global.ca.v4.u32 { %r10412,…},[%rd126]
|
||||
```
|
||||
|
||||
| epilogue | range | shared read (offset) | becomes | B offsets | next phase's W block |
|
||||
|---|---|---|---|---|---|
|
||||
| E11 | s10690–s10712 | s10693, `+2048` | P12 A | 45312 … 48896 step 512 (8 loads) | P12 `45312..48908` |
|
||||
| E12 | s10729–s10751 | s10732, `+4096` | P13 A | 69888 … 73472 | P13 `69888..73484` |
|
||||
| E13 | s10768–s10790 | s10771, `+6144` | P14 A | 94464 … 98048 | P14 `94464..98060` |
|
||||
|
||||
So phases 11…14 (which all have `nA = 1`) each take their single 16×32 A fragment
|
||||
from one 512-byte slot of the shared arena and their B from `W`.
|
||||
|
||||
### E14 — s10807–s11638: the RMS norm before the MLP
|
||||
|
||||
832 statements: 242 `mov.b32`, 192 `mul.f16`, 66 `add.f16`, 64 `cvt.f32.f16`,
|
||||
64 `rsqrt.approx.ftz.f32`, 64 `cvt.rn.f16.f32`, 34 `add.f16x2`, 4
|
||||
`shfl.sync.bfly`, 32 packs, 16 `ld.global.v2.u16`, 8 weight `v4.u32`, 4
|
||||
`ld.global.u32`.
|
||||
|
||||
| range | what |
|
||||
|---|---|
|
||||
| s10810–s10935 | 64 `mul.f16` — squares of phase 14's 32 D f16x2 (`s10809 L23500: mov.b32 {%rs4566,%rs4569}, %r10832`, `s10810 mul.f16 %rs3760,%rs4569,%rs4569`) |
|
||||
| s10937–s10999 | 34 `add.f16x2` — in-lane reduction, with the 4 `shfl.sync.bfly` over lane bits 0,1 at s10971/L23947, s10977, s10983, s10989 |
|
||||
| s10996–s11001 | `add.f16x2` + `add.f16` — the two row sums per lane |
|
||||
| s11002–s11097 | `+ 2⁻¹³` (`s11003 add.f16 %rs3959,%rs3949,%rs3955`, `%rs3955 = cvt.rn.f16.f64 %fd1` s11002), broadcast into 32 f16x2 |
|
||||
| s11102–s11351 | 64 `rsqrt.approx.ftz.f32` (2 halves each) → 32 reciprocals |
|
||||
| s11355–s11370 | 16 `ld.global.v2.u16` gain at `W + ((laneid<<2)&12) + {98816 + 16*i}` |
|
||||
| s11372–s11466 | 64 `mul.f16` — `inv · gain` (e.g. `s11372 L24749: mul.f16 %rs4148,%rs4149,%rs5458` with `%rs5458 = W+98816`) |
|
||||
| s11467–s11561 | 64 `mul.f16` — `x · (inv·gain)` (`s11468 mul.f16 %rs4340,%rs4566,%rs4148`) |
|
||||
| s11567–s11581 | 8 weight `v4.u32` = phase 15's B (`99072 + 512*j`) |
|
||||
| s11587–s11590 | 4 `ld.global.u32` at `W + ((laneid<<2)&12) + {103168,103184,103200,103216}` = phase 15's C bias |
|
||||
| s11591–s11637 | 32 packs of the normalised values → phase 15's A |
|
||||
|
||||
**What it computes.** One RMS reduction per **row** over all 128 columns, with
|
||||
no mean subtraction and no `1/N`: the lane's 16 one-per-n-tile partials for that
|
||||
row are tree-summed in-lane (s10937–s10965), the 4 lanes holding the four
|
||||
`t = laneid&3` column pairs are folded with two butterfly steps, and the f16x2
|
||||
halves are added (`s10998 add.f16 %rs3949,%rs3950,%rs3951`). Then
|
||||
|
||||
```
|
||||
inv = f16(rsqrt.approx(f32(f16(sumsq + 2^-13))))
|
||||
out[row,c] = x[row,c] · inv · gain[c]
|
||||
```
|
||||
|
||||
32 `rsqrt` results are produced because the compiler re-emits the reciprocal once
|
||||
per (row, n-tile) pair, not because there are 32 distinct sums: the 32 `mov.b32`
|
||||
that feed them copy only **two** distinct halves (`%rs3949` row `g`, `%rs3952` row
|
||||
`g+8`, e.g. `s11005 mov.b32 %r11121, {%rs3956, %rs3959}` where `%rs3956 = %rs3959
|
||||
= %rs3949 + 2^-13`).
|
||||
|
||||
`gain[c]` is per column and constant across rows (the four lanes with equal
|
||||
`laneid&3` read the same 16-byte record). **The norm's bias is not applied
|
||||
here** — it is phase 15's C operand, loaded at the end of this epilogue
|
||||
(`+103168 … +103216`).
|
||||
|
||||
**Outputs:** phase 15's B (8 regs) and C (4 regs), 16 packed A registers, and the
|
||||
32 normalised f16x2 registers that E15/E17/E19 re-read as `%rs4566 …`.
|
||||
|
||||
### E15 — s11655–s11943: clamped cubic activation + column table + pack
|
||||
|
||||
289 statements: `mov.b32`×68, `add.f16`×64, `cvt.rn.f16.f32`×32, `mul.f16x2`×24,
|
||||
16 `ld.global.v2.u16`, 8 each `neg/max/min/abs/sub/add.f16x2`, 8 weight `v4.u32`,
|
||||
8 packs, `mov.f32`×4.
|
||||
|
||||
**(a) the constants and the activation, s11656–s11795.** Eight groups, each
|
||||
materialising `%f989…%f992` (`s11656 L25598: mov.f32 %f989, 0f3ED306EB`, `s11659
|
||||
mov.f32 %f990, 0f3DA60DD6`, `s11662 mov.f32 %f991, 0f3F000000`, `s11665 mov.f32
|
||||
%f992, 0f40000000`) and then doing, on **phase 15's D**
|
||||
(`%r11311 %r11321 …`):
|
||||
|
||||
```
|
||||
y = clamp(x, -2, +2) // s11668 neg, s11669 max, s11670 min
|
||||
g = 0.5 + y*(0.412109375 - 0.0810546875*|y|) // s11671 abs, s11672 mul, s11673 sub,
|
||||
// s11674 mul, s11675 add
|
||||
out = x * g // s11676 mul.f16x2
|
||||
```
|
||||
|
||||
`g` is the same cubic ramp as enc0's E13 (`g(+2) = 1`, `g(-2) = 0`), **not** a
|
||||
sigmoid/SiLU. 8 groups cover the 8 register-pairs of the 4 chains of P15.
|
||||
|
||||
**(b) column bias, s11815–s11931.** 16 `ld.global.v2.u16` at
|
||||
`W + ((laneid<<2)&12) + {107328 + 16*i}` then 64 `add.f16`, e.g.
|
||||
`s11837 L26147: add.f16 %rs4564,%rs5490,%rs4566` — the second addend is a half of
|
||||
phase **14**'s D (`%rs4566/%rs4569` ← `%r10832`, s10809), i.e. a residual. The
|
||||
48 results `%r11694 … %r11845` are **phase 16's C operand**.
|
||||
|
||||
**(c) pack, s11932–s11943.** 8 packs of the **activation outputs**
|
||||
(`s11932 cvt %rs4757, %r11554`, `s11933 cvt %rs4756, %r11438`, `s11934 mov.b32
|
||||
%r11838, {%rs4756,%rs4757}`, …) → phase 16's A fragment (`s11944
|
||||
A={%r11838,%r11839,%r11840,%r11841}`).
|
||||
|
||||
**(d) next B, s11800–s11814.** 8 `v4.u32` at `103232 + 512*j` → phase 16's B.
|
||||
|
||||
**No stores, no `ld.global` other than the two tables.** Inputs: phase 15's D,
|
||||
phase 14's D, `W`. Outputs: phase 16's A (4 regs), B (32), C (32).
|
||||
|
||||
### E16 / E18 / … / E44 — s11960–s12035 and its 14 repeats: next B/C loads + pack
|
||||
|
||||
76 statements, identical shape, for every even phase 16…44:
|
||||
|
||||
```
|
||||
s11964 L26577: ld.weak.global.ca.v4.u32 { %r11847,…},[%rd166] // B 103232 + 512*j, 8 loads
|
||||
s11984 L26614: ld.global.u32 %r11921, [%rd455+111680] // C bias, 4 words at +111680+16*i
|
||||
s11988 L26619: cvt.rn.satfinite.e4m3x2.f16x2 %rs…, %r… // 32 packs
|
||||
s11990 L26624: mov.b32 %r…, {%rs…, %rs…} // 16 b32 = 4 A fragments
|
||||
```
|
||||
|
||||
The 32 pack sources (`%r21120 %r21121 %r21122 …`, `s11988 L26619: cvt
|
||||
%rs4765, %r21121`) are the **normalised values** E14 produced with the second of
|
||||
its multi-sets (`s11469 mov.b32 %r21120, {%rs4340, %rs4343}`), i.e. phase 17's A
|
||||
operand is the *same* normed tile that phase 15's A was built from — phases 15
|
||||
and 17 are two parallel GEMMs off one normed tile, each with its own weight block
|
||||
and bias. The 4 C words are `W + ((laneid<<2)&12) + {bias_base + 16*i}`, and
|
||||
each is used in **both halves** of the *odd* phase's mma C operand:
|
||||
`s12036 C={%r11921,%r11921}` with `[%r11921=W+111680]`, `s12037
|
||||
C={%r11931,%r11931}` with `W+111696`.
|
||||
|
||||
| epilogue | range | next-B block | next-C bytes |
|
||||
|---|---|---|---|
|
||||
| E16 | s11960–s12035 | 103232..106828 | 111680..111728 |
|
||||
| E18 | s12236–s12311 | 111744..115340 | 119936..119984 |
|
||||
| E20 | s12512–s12587 | 120000..123596 | 128192..128240 |
|
||||
| E22 | s12788–s12863 | 128256..131852 | 136448..136496 |
|
||||
| E24 | s13064–s13139 | 136512..140108 | 144704..144752 |
|
||||
| E26 | s13340–s13415 | 144768..148364 | 152960..153008 |
|
||||
| E28 | s13616–s13691 | 153024..156620 | 161216..161264 |
|
||||
| E30 | s13892–s13967 | 161280..164876 | 169472..169520 |
|
||||
| E32 | s14168–s14243 | 169536..173132 | 177728..177776 |
|
||||
| E34 | s14444–s14519 | 177792..181388 | 185984..186032 |
|
||||
| E36 | s14720–s14795 | 186048..189644 | 194240..194288 |
|
||||
| E38 | s14996–s15071 | 194304..197900 | 202496..202544 |
|
||||
| E40 | s15272–s15347 | 202560..206156 | 210752..210800 |
|
||||
| E42 | s15548–s15623 | 210816..214412 | 219008..219056 |
|
||||
| E44 | s15824–s15899 | 219072..222668 | 227264..227312 |
|
||||
|
||||
### E17 / E19 / … / E45 — s12052–s12219 and its 15 repeats: clamped cubic activation + pack
|
||||
|
||||
168 statements; identical op mix to E15 minus the table loads and the `mov.f32`
|
||||
block (36 `mov.b32`, 32 `cvt.rn.f16.f32`, 24 `mul.f16x2`, 8 each
|
||||
`neg/max/min/abs/sub/add.f16x2`, 8 weight `v4.u32`, 8 packs):
|
||||
|
||||
```
|
||||
s12061 L26870: neg.f16x2 %r…,%r…
|
||||
s12062 L26874: max.f16x2 %r…,%r11912,%r…
|
||||
s12065 L26886: mul.f16x2 %r…,%r…,%r…
|
||||
s12193 L27332: ld.weak.global.ca.v4.u32 { …},[%rd174] // next B, 8 loads
|
||||
s12208 L27363: cvt.rn.satfinite.e4m3x2.f16x2 … // 8 packs of activation outputs
|
||||
```
|
||||
|
||||
Same function as E15(a) on the odd phase's D (`%r11912 %r11922 …`), 8 packs of the
|
||||
activation *outputs* → the next even phase's A, and 2 weight groups → the next even
|
||||
phase's B. **No column bias, no `ld.global`.** The 15 repeats are at
|
||||
s12052, s12328, s12604, s12880, s13156, s13432, s13708, s13984, s14260, s14536,
|
||||
s14812, s15088, s15364, s15640, s15916 (one per odd phase 17…45).
|
||||
|
||||
#### Shape of the attention + MLP region, E10–E46
|
||||
|
||||
```
|
||||
P9 A = pack(P7.D) B = pack(P7.D) C = bias@8448+512i → scores 64×96
|
||||
E9: cubic-exponent softmax (no row max), 48 movmatrix(P8.D) + packs
|
||||
P10 A = pack(scores) B = pack(transpose(P8.D)) C = 0 → 64×32
|
||||
E10: pack → shared slot, read back as P11 A; table@98560 + residual → P11 C
|
||||
P11 A = shared, B = W@20736, C = table+residual → D11
|
||||
E11: read shared slot +2048 → P12 A; load P12 B
|
||||
P12 A = shared, B = W@45312, C = D11 → D12 E12: same for P13
|
||||
P13 A = shared, B = W@69888, C = D12 → D13 E13: same for P14
|
||||
P14 A = shared, B = W@94464, C = D13 → D14 (16×128)
|
||||
E14: row-RMS norm over 128 columns (+2^-13, rsqrt, gain@98816), pack → P15 A
|
||||
P15 A = normed tile, B = W@99072, C = bias@103168 → D15 (banded 64×64×128)
|
||||
E15: cubic activation(D15); table@107328 + D14 → P16 C; pack → P16 A
|
||||
P16 A = pack(activation), B = W@103232, C = D14+table → D16
|
||||
E16: pack of the *normed* tile → P17 A; load P17 B and C
|
||||
P17 A = normed tile, B = W@107584, C = bias@111680 → D17
|
||||
…the P15/P16 pair then repeats 15 more times (P17/P18 … P45/P46),
|
||||
each odd phase with its own weight block + bias and each even phase
|
||||
re-using the same normed tile as A and the previous even phase's D as C.
|
||||
P46 A = pack(activation P45), B = W@227328, C = D44 → D46 (16×128)
|
||||
E46: arena scatter, shared staging, next B/C
|
||||
P47 A = 16 shared slots, B = W@231424, C = bias@313344 → 4×16 k-steps
|
||||
E47: pack + surface scatter to param+56, next B/C
|
||||
P48 A = pack(E47), B = W@313664, C = bias@315712 → D48
|
||||
E48: the final epilogue
|
||||
```
|
||||
|
||||
### E46 — s16100–s16663: pack → plane-arena scatter → shared staging → next B/C
|
||||
|
||||
564 statements. Histogram: `add.s32`×66, `ld.shared.u32`×64, `add.s64`×53,
|
||||
`setp.lt.s32`×36, `shl.b32`×35, `bra`×33, `cvt.rn.satfinite.e4m3x2.f16x2`×32,
|
||||
`selp.b32`×32, `ld.weak.global.ca.v4.u32`×32, `mov.b32`×16, `mul.wide.s32`×16,
|
||||
`st.global.u32`×16, `st.shared.u32`×16, `and.b32`×11, `mov.u32`×10, `shr.u32`×10,
|
||||
`or.b32`×9, `setp.ge.u32`×4, `ld.global.u32`×4, `ld.param.u64`×2, `bar.sync`×2.
|
||||
|
||||
**(a) packs, s16100–s16147.** 32 `cvt.rn.satfinite.e4m3x2.f16x2` + 16 `mov.b32`
|
||||
→ `%r238 … %r253` = phase 46's 16×128 output as 16 b32 = 32 e4m3 per lane.
|
||||
|
||||
**(b) plane-arena scatter, s16148–s16326.** `%rd4 = cvta(ld.param.u64 [%rd1+-32])`
|
||||
(s16148–s16149, `%rd1 = param_0+80` ⇒ **`param_0+48`**), lane decomposition
|
||||
`s16151 shr %r254, laneid, 2` (`g`), `s16152 and %r255, laneid, 3` (`t`), and 8
|
||||
index bases formed from `(%r1 + ((g+8j)&7))`, `(%r6' + ((g+8j)>>3))` and
|
||||
`(%r255 | 4)`, each guarded by `setp.lt.s32 %p, idx, 0` + `bra`:
|
||||
|
||||
| store | value | index |
|
||||
|---|---|---|
|
||||
| s16174 / L40691 | `%r238` | `t + 4*(g&7) + 32*(y)` |
|
||||
| s16183 / L40702 | `%r239` | same, `t` or 4 |
|
||||
| s16216 / L40737 | `%r240` | second base (`y2 = tid.z*2 + ((g+8)>>3)`) |
|
||||
| s16224 | `%r241` | `t` or 4 |
|
||||
| s16233 / s16241 | `%r242`, `%r243` | `+512` of the two bases |
|
||||
| s16250 / s16258 | `%r244`, `%r245` | |
|
||||
| s16267 / s16275 | `%r246`, `%r247` | `+1024` |
|
||||
| s16284 / s16292 | `%r248`, `%r249` | |
|
||||
| s16301 / s16309 | `%r250`, `%r251` | `+1536` |
|
||||
| s16318 / s16326 | `%r252`, `%r253` | |
|
||||
|
||||
The exact index algebra is in §6/U4 — the *base* splits as
|
||||
`t + 4*(g&7) + 32*y` with `y = (tid.z<<1) + (laneid>>5)` for the first group and
|
||||
`y = (tid.z<<1) + ((g+8)>>3)` for the second, and the second group's `+512`
|
||||
strides. What is certain: 16 guarded 4-byte stores, i.e. `8*32 = 256` bytes per
|
||||
row-stride unit of the arena.
|
||||
|
||||
**(c) shared staging, s16330–s16480.** 16 `st.shared.u32` at
|
||||
`%r22114 = %r4244 + (idx<<2)` with `idx = t | ((laneid&28) | (y<<5))` and
|
||||
`y = (tid.z<<1) + (laneid>>5)`, then `+256` and `+512*j`, `j = 0…7`
|
||||
(`s16348 L40899: st.shared.u32 [%r22114], %r238`, `s16356 [+256] …`). A
|
||||
`bar.sync 0` at s16485 separates the two halves.
|
||||
|
||||
**(d) staging read-back, s16486–s16574.** 64 `ld.shared.u32` at
|
||||
`%r4244 + (idx2<<2)` with
|
||||
`idx2 = ((laneid&12)<<1) | ((laneid&~15)<<2) | (laneid&3) (= 16*g + t)` for the
|
||||
first 32 and `idx2 + 128` for the second 32, at byte offsets `{0, 1024, 2048,
|
||||
3072, 4096, 5120, 6144, 7168}` (`s16502 L41087: ld.shared.u32 %r304, [%r22196]`,
|
||||
`s16503 [+1024]`, `s16503–s16522`, then `s16527 [+16] … s16542`). These 64 values
|
||||
`%r304 … %r367` are **phase 47's 16 A fragments**.
|
||||
|
||||
**(e) next B/C, s16581–s16663.** Inside a loop `$L__BB1_324` (`s16582`),
|
||||
`s16583 shl.b32 %r22998, %r23413, 9` then 32 `v4.u32` at
|
||||
`W + 16*laneid + 231424 + 512*(%r23413 + j)`, `j = 0…31` (`s16590 – s16652`) =
|
||||
phase 47's 64 B fragments, and 4 `ld.global.u32` at
|
||||
`W + ((laneid<<2)&12) + {313344, 313360, 313376, 313392}` (s16660–s16663) =
|
||||
phase 47's C bias.
|
||||
|
||||
**Outputs:** `%r238…%r253` (read again by (c)), phase 47's A/B/C.
|
||||
|
||||
### E47 — s16728–s16852: pack → surface scatter → next B/C
|
||||
|
||||
125 statements: 8 packs + 2 `mov.b32` (s16728–s16737), 4 guarded stores to
|
||||
**`param_0+56`** (s16772 `st.global.v2.u16`, s16783 `st.global.u32`,
|
||||
s16817 `st.global.u32`, s16826 `st.global.v2.u16`), then 4 weight `v4.u32`
|
||||
(s16839–s16845) and 2 `ld.global.u32` (s16851–s16852).
|
||||
|
||||
```
|
||||
s16738 L41793: mov.u32 %r22997, %laneid
|
||||
s16744 L41800: shr.u32 %r23004, %r22183, 1 // ey>>1
|
||||
s16745 L41801: shr.u32 %r23005, %r22182, 1 // ex>>1
|
||||
s16746 L41802: shl.b32 %r23006, %r23005, 3 // 8*(ex>>1)
|
||||
s16747 L41803: mul.lo.s32 %r23007, %r23004, %r23006 // row stride
|
||||
s16769 L41826: cvta.to.global.u64 %rd632, %rd5 // %rd5 = param_0+56 (s16581)
|
||||
s16772 L41829: st.global.v2.u16 [%rd634], {%rs5492, %rs5493}
|
||||
```
|
||||
|
||||
The destination is `param+56`, **not** the plane arena; extents come from
|
||||
`ld.param.v2.u32 [param_0+0]` re-read as `%r22182/%r22183` in E46 (s16487) and are
|
||||
**halved**. Two index bases are formed, each guarded by
|
||||
`(y<0)|(y≥(ey>>1))|(laneid>63)|(x<0)|(x≥(ex>>1))|(%r23413>159)`: base 1
|
||||
`y = (%r2>>1) + (laneid>>4)`, `x = (%r1>>1) + ((laneid>>2)&3)`; base 2 uses
|
||||
`(g+8)`. The loop at s16828–s16830 steps `%r23413` by 128 while `< 160`, so the
|
||||
8-pack/4-store body runs once or twice depending on `tid.z` (§6/U4).
|
||||
|
||||
Then phase 48's B (4 `v4.u32` at `313664 + 512*j`) and C (2 `ld.global.u32` at
|
||||
`315712`, `315728`).
|
||||
|
||||
### E48 — s16861–s17183: the final epilogue
|
||||
|
||||
323 statements: 4 `st.shared.u32` (s16872–s16891), 2 `bar.sync`, 5 `ld.shared.u32`
|
||||
(s16930–s16938), 12 `mul.ftz.f32`/`ex2.approx`/`add.ftz.f32`/`div.approx`,
|
||||
4 `shfl.sync.down.b32`, 9 `tex.base.2d.v4.f16.s32` (s17042–s17101), 3
|
||||
`sust.b.2d.v4.b16.zero` (s17165, s17166, s17181).
|
||||
|
||||
**(a) shared staging, s16861–s16891.** Phase 48's D (`%r23106/%r23107` and
|
||||
`%r23116/%r23117`) is written to a shared array at
|
||||
`%r391 = %r23368 + 4*(6*((laneid>>2) + (tid.z<<4) ... ) + t)` and `%r393` (the
|
||||
`(g+8)` variant), two `u32` per lane with a `%r392 > 5` guard on the second.
|
||||
|
||||
**(b) row max, then softmax, s16904–s17022.** After `bar.sync 0` (s16904)
|
||||
`%tid.x` picks a 6-element row inside a shared array
|
||||
(`s16930–s16936: ld.shared.u32 [%r23313] … [+20]`, with `%r23313 = %r23380 +
|
||||
4*(6*x + 48*y)`), and:
|
||||
|
||||
```
|
||||
s16948 L42087: max.f16x2 %r23163,%r23164,%r23165 // 5-element row max
|
||||
s16952 L42100: max.f16 %rs5502,%rs5503,%rs5504
|
||||
s16954 L42107: mov.b32 %r23186, {%rs5505, %rs5505}
|
||||
s16955 L42109: sub.f16x2 %r23172,%r23164,%r23186 // x - rowmax
|
||||
s16959 L42118: ex2.approx.ftz.f32 %f997, %f1025 // 2^((x-max)*log2e)
|
||||
s17015 L42237: div.approx.ftz.f32 %f1013, %f1024, %f1042 // 1/rowsum
|
||||
s17018 L42244: mul.f16x2 %r23187,%r23188,%r23201
|
||||
```
|
||||
|
||||
**This one has a real row maximum** (unlike E9). The `ex2` is a *true* `2^u` via
|
||||
`mul.ftz.f32` by `0f3FB8AA3B` (= log2 e) plus `ex2.approx.ftz.f32`, not the cubic
|
||||
mantissa trick. Note the extra `add.ftz.f32 %f1028, %f1027, 0f00000000` at
|
||||
s16967 (a zero add the compiler left in).
|
||||
|
||||
**(c) texture gather + weighted sum, s17023–s17105.** A 3×3 spatial stencil of
|
||||
`tex.base.2d.v4.f16.s32` reads from the texture at `param_0+88`
|
||||
(`s16937 ld.param.u64 %rd672, [%rd681+112]` is a surface; the texture is
|
||||
`s17040 ld.param.u64 %rd670, [%rd681+88]`), with the 8 neighbour coordinates
|
||||
clamped by `selp` chains (`s17024–s17035`: `x-1`, `x+1`, `y-1`, `y+1`, both with
|
||||
`1` as the out-of-range sentinel). Each 3×3 texel is multiplied by one of the
|
||||
five softmax values with `fma.rn.f16x2` (`s17046 fma.rn.f16x2 %r23206,%r23207,
|
||||
%r23187,%r23217` with `%r23217 = 0`), the halves are summed
|
||||
(`s17098 add.f16 %rs5559,%rs5560,%rs5561`) and one more texel is folded with
|
||||
`fma.rn.f16` (s17103–s17105).
|
||||
|
||||
**(d) second sigmoid and difference, s17106–s17161.** Two more
|
||||
`1/(1+2^(-log2e·v))` reciprocals (`s17107–s17111`, `s17113–s17117`), three
|
||||
`mul.f16` products (`s17118–s17120`) and three `sub.f16` differences
|
||||
(`s17121–s17124`), the `0.25` (`s17122 mov.f32 %f1020, 0f3E800000`) scale
|
||||
(s17129 `mul.f16x2 %r23268, %r23269, %r23270`), then a **two-step cross-lane
|
||||
sum**: `shfl.sync.down.b32` with `delta = %r23293 = 1` (s16940) and
|
||||
`delta = %r23301 = 8` (s16933), each followed by `add.f16x2`, and the same two
|
||||
steps again on the scalar path (s17149 with delta 1, s17158 with delta 8, then
|
||||
`add.f16` at s17152 and s17161). The clamp/segment operand is again
|
||||
`(WARP_SZ<<8)-8192 | 31` and the membermask `%r23303 = -1` (s17026).
|
||||
|
||||
**(e) surface stores, s17162–s17183.**
|
||||
|
||||
```
|
||||
s17162 L42567: ld.param.u64 %rd674, [%rd681+120]
|
||||
s17163 L42568: ld.param.u64 %rd677, [%rd681+128]
|
||||
s17165 L42571: sust.b.2d.v4.b16.zero [%rd672, {%r23307,%r397}], {%rs5597,%rs5600,%rs5603,%rs5634}
|
||||
s17166 L42574: sust.b.2d.v4.b16.zero [%rd674, {%r23307,%r397}], {%rs5588,%rs5591,%rs5594,%rs5635}
|
||||
s17181 L42592: sust.b.2d.v4.b16.zero [%rd677, {%r23335,%r23336}], {%rs5636,%rs5637,%rs5621,%rs5639}
|
||||
```
|
||||
|
||||
Three 4-channel b16 surface writes (`param_0+112`, `+120`, `+128`); the third is
|
||||
taken only for odd `%r398 & 1` (`s17167 and.b32 %r23334, %r398, 1`, `s17169
|
||||
mov.pred %p347, 0`, `s17170 xor.pred %p348, %p346, %p347`, s17171 `@%p348 bra`) and
|
||||
stores `0` in its fourth channel (`s17180 mov.u16 %rs5639, 0`).
|
||||
|
||||
**Inputs:** phase 48's D, the 18 cut values, `param+0/+88/+112/+120/+128`, the
|
||||
shared staging of (a) and the normed tile's leftovers.
|
||||
**Outputs:** none (the kernel `ret`s at s17183 / L42596).
|
||||
|
||||
## 4. The plane-arena and surface writes
|
||||
|
||||
`param_0+48` is reached once (`s16148 ld.param.u64 %rd556, [%rd1+-32]` with
|
||||
`%rd1 = param_0+80` at s4) and is never the base of an `ld.*` in this entry: the
|
||||
arena is write-only within a launch. The complete set of `st.global.*` /
|
||||
`sust.b.*` in the entry is:
|
||||
|
||||
* **E46, 16 × `st.global.u32 [%rd4 + 4*idx], v`** — s16174, s16183, s16216,
|
||||
s16224, s16233, s16241, s16250, s16258, s16267, s16275, s16284, s16292, s16301,
|
||||
s16309, s16318, s16326 (§3/E46(b)). Value: phase 46's requantised 16×128
|
||||
output, 16 b32 per lane.
|
||||
* **E47, 4 stores to `param_0+56`** — s16772 (`v2.u16`), s16783 (`u32`),
|
||||
s16817 (`u32`), s16826 (`v2.u16`).
|
||||
* **E48, 3 `sust.b.2d.v4.b16.zero`** — s17165, s17166, s17181 to surfaces at
|
||||
`param_0+112/+120/+128`.
|
||||
|
||||
Nothing else in the 17183-statement entry writes to global memory.
|
||||
|
||||
## 5. Uncharacterised
|
||||
|
||||
**U1. The identity of the 11792-byte score bias (P9's C operand).** Phase 9 adds
|
||||
the f16 table `[8448, 20236)` indexed as
|
||||
`byte = 8448 + 512*(4*(n>>1) + m) + 16*laneid + 8*(n&1)` (verified against phase
|
||||
9's own C operands: `s6536 C=%r6280,%r6281` ↔ `i=0`, `s6538 C=%r6296,%r6297` ↔
|
||||
`i=4`, both loaded by `s6369`/`s6377`). Whether it is a relative position bias, a
|
||||
learned score bias or a per-lane constant cannot be decided from the PTX.
|
||||
*Missing:* the weight image's layout description (the `_prep` permutation).
|
||||
|
||||
**U2. The identity of the per-column tables.** `W+98560` (E10's C seed add),
|
||||
`W+98816` (E14's norm gain), `W+107328` (E15's post-activation add) and
|
||||
`W+103168`/`111680`/… (the odd phases' GEMM bias) are all read through the same
|
||||
`W + ((laneid<<2)&12) + off` pattern. Their *role* is computable (see §3) but
|
||||
their model names are not.
|
||||
|
||||
**U3. The shared-arena layout.** The arena is written by the prologue
|
||||
(s6054–s6083), by E10 (s10491–s10530) and by E46 (s16348–s16480), and read by
|
||||
P1–P8 (s6090–s6113), P11–P14 (s10537, s10693, s10732, s10770) and P47
|
||||
(s16502–s16574). The prologue writes at `%r235 + 16*laneid + {0,512,1024,1536}`
|
||||
and `%r235 + 16*laneid + {8192…9728}` (s6058, s6068–s6083) while the P1–P8 reads
|
||||
use `%r4244 + 16*laneid + 512*k` with **no** `tid.z` term (s6090 ff.) — so at
|
||||
least one of the two index derivations is not reconstructible without the
|
||||
`_prep`/launch geometry. *Missing:* the launch's `%tid.z`/`ctaid` ranges, or a
|
||||
memory dump.
|
||||
|
||||
**U4. E46's/E47's index algebra.** E46(b)'s arena stores and E47's
|
||||
`param+56` stores each form two index bases from `(laneid&28) | (y<<5)` and
|
||||
`((g+8)<<2)&28 | (y2<<5)` plus `+512*j`, with `y`/`y2` differing by one in the
|
||||
row (they are derived from `%r23351` and `%r23353`, both `tid.z<<1`); the shared
|
||||
staging read in E46(d) uses a *different* index
|
||||
(`16*g + t` and `16*g + t + 128`). I could not prove the store↔load permutation
|
||||
from the PTX alone, nor whether `%r23413`'s loop in E47 (s16828–s16830) executes
|
||||
once or twice per `tid.z` (the guard `%r23413 > 159` at s16489/s16743 is checked
|
||||
before the first body). *Missing:* a run of the kernel with a plane-arena dump.
|
||||
|
||||
**U5. The MLP block's shape.** §3 establishes, from the pack sources and each
|
||||
phase's C operand, that the MLP region is: one normed tile feeding every *odd*
|
||||
phase (P15, P17, …, P45) as its `A`, each odd phase's activation feeding its
|
||||
*paired* even phase as `A`, and the even phases' `C` chaining (`D16 = … + D14 +
|
||||
table@107328`, `D18 = … + D16`, …, verified from `P18 C` tracing to P16's mma).
|
||||
Only the **first** even phase adds the column table `@107328` to the residual
|
||||
(s11837 versus the table-free `E18`/`E20`/…); every later even phase's `C` is the
|
||||
previous even phase's `D` alone (E18's `C` operands have no `ld.` in their def-use
|
||||
chain). Whether the 16 pairs are one wide MLP, a stack of residual blocks, or
|
||||
something else is a model-level question the PTX cannot answer.
|
||||
*Missing:* the model source.
|
||||
|
||||
**U6. The stem's shift structure.** Phases 1…8 read the same 24 shared A
|
||||
fragments in four shift groups (P1/P2 at `+0/1536/…`, P3/P4 at `+512/2048/…`,
|
||||
P5/P6 at `+1024/…`, P7/P8 at `+1536/…`, s6090–s6113) and accumulate in two
|
||||
chains (`P1→P3→P5→P7` and `P2→P4→P6→P8`, verified from each phase's C operands:
|
||||
`s6186 C={%r4296,%r4297}` = P1's D). Whether that is a 2×2 stencil, a 4-tap
|
||||
1-D convolution, or something else is a property of the high-level model, not of
|
||||
the PTX. (enc4 has five shift groups and ten phases, so the tap count is a
|
||||
per-stage constant.) *Missing:* the model source.
|
||||
|
||||
**U7. The meaning of the prologue residual vector.** §1.2 tracks the f16 halves
|
||||
`%rs1460 …` back to `cvt.rn.f16x2.e4m3x2` of e4m3 values loaded from the plane at
|
||||
`param_0+8` (`s175`, `s480`, …). What that plane holds (an input colour buffer,
|
||||
a noise texture, a previous frame) is not determinable from the PTX.
|
||||
|
||||
**U8. The logical meaning of E48's final expression.** The tail computes
|
||||
`out = (acc − sigmoid(acc)) * 0.25` on three channels gathered by texture from a
|
||||
3×3 stencil (same shape as enc0's E22(e), whose §6/U6 flags the same gap).
|
||||
|
||||
## 6. Quick acceptance index
|
||||
|
||||
| requirement | where |
|
||||
|---|---|
|
||||
| every phase's M/N/K with the arithmetic checked | §2 (rows flagged **N** are banded; the real shapes are derived there) |
|
||||
| weight image extent | §2 (315732 bytes, max offset 315728 at s16852) |
|
||||
| phases that read no weight operand | §2 (P9, P10) |
|
||||
| score epilogue: row max? | **none** — §3/E9, 16 `shfl.sync.bfly` only, no `max` used for a maximum |
|
||||
| score bias operand | §3/E8 (24 `v4.u32` at `W+8448+512*i`, indexed `4*(n>>1)+m`) |
|
||||
| attention Q/K/V | §3/E8 (P9 A and B both from P7's D), §3/E9(d) (P8's D transposed) |
|
||||
| the norm | §3/E14 (row RMS over 128 columns, `+2^-13`, `rsqrt.approx`, gain) |
|
||||
| the activations | §3/E15, §3/E17 (clamped cubic ramp, 8 groups per phase) |
|
||||
| patch merge | §3/E46 (arena + shared staging), §3/E47 (`param+56`), §3/E48 (textures + surfaces) |
|
||||
| per-launch plane-arena writes | §4 |
|
||||
Reference in new issue
Block a user