woutermans
0fe3a28cb7
Per-layer staged-image engine for the eleven translated denoiser Swin layers: descriptors (verified offsets per phase), cut recipes that splice an isolated native phase into NVIDIA's PTX, wave templates, epilogue notes and GPU probes. The isolated phase-8 splice produces a byte-identical final 4K frame but is slower than the translated body (14.90 vs 6.59 ms) - a correctness experiment, not a shipping improvement; docs/rr-engine.md records the design and the next direct-converter win (B operands built from transposed accumulators).
1007 lines
51 KiB
Markdown
1007 lines
51 KiB
Markdown
# `cuda_dldn_engine_swin_dec1_kernel` — the epilogues of phases 1…26
|
||
|
||
Companion to `kernels/rr/rr_layer_spec.py dec1`. Everything below was read out of
|
||
`~/.cache/d4r-rr-corpus/0022-PREPASS_ENTRYPOINT_NAME.ptx`, entry
|
||
`cuda_dldn_engine_swin_dec1_kernel` (16304 statements, 44528 physical lines).
|
||
|
||
## 0. Conventions
|
||
|
||
* **`sNNNN` is a statement index** in the linearisation `rr_layer_spec.py`
|
||
documents (statements of the entry body; a wrapped `{ … }` block and a 4-line
|
||
`mma.sync` count as one statement, numbered at their first line). **`LNNNNN`
|
||
is the physical line** of the same instruction in the corpus file, so every
|
||
claim can be checked with `sed -n 'LNNNNN,LNNNNN p' …ptx`.
|
||
* An **epilogue** is the run of statements between one phase's last `mma` and
|
||
the next phase's first `mma`. `rr_layer_spec.py dec1` prints the ranges; this
|
||
file names what each one *computes*. Because phase 1 is the first GEMM, the
|
||
prologue (s1–s402) doubles as phase 1's prelude and is described here too.
|
||
* `W` denotes the prepared weight image: `ld.param.u64 %rd2,
|
||
[cuda_dldn_engine_swin_dec1_kernel_param_0+40]` (s8, `L1036`) and
|
||
`cvta.to.global.u64 %rd3, %rd2` (s9, `L1037`). Every
|
||
`ld.weak.global.ca.v4.u32` reads `W + 16*laneid + imm`.
|
||
* **Byte offsets are relative to `W` and (where noted) carry a `tid.z` term.**
|
||
Two epilogues' load chains add a per-`tid.z` stride that the rest of the entry
|
||
does not, so the offsets in the phase table are *not* all on one base — §1.1
|
||
and §7/U6.
|
||
* The C/D register layout and the weight-addressing formula are fixed (see the
|
||
settled notes): `C/D reg0 = (row g, cols 2t,2t+1)`, `reg1 = (row g+8, same
|
||
cols)`, `g = laneid>>2`, `t = laneid&3`.
|
||
* Anything I could not pin down from the PTX is in §7 under **uncharacterised**.
|
||
|
||
## 1. Shared vocabulary
|
||
|
||
### 1.1 Addresses and parameters an epilogue touches
|
||
|
||
| symbol | how it is obtained | statements |
|
||
|---|---|---|
|
||
| `param_0` | `mov.b64 %rd8, …_param_0` | s1, L1018 |
|
||
| `W` (weight image) | `ld.param.u64 %rd2, [param_0+40]` → `cvta… %rd3, %rd2` | s8/s9, L1036–L1037 |
|
||
| extents `(ex, ey)` | `ld.param.v2.u32 {%r2195, %r2196}, [param_0+0]` | s14, L1042 |
|
||
| `%r7 = 8·ex`, `%r8 = ey·8·ex` | `shl.b32 %r7, %r2195, 3` / `mul.lo.s32 %r8, %r2196, %r7` | s15/s16, L1043–L1044 |
|
||
| block origins `%r1`, `%r2` | `sub.s32 %r1, %r2190, %r2191` / `sub.s32 %r2, %r2194, %r2192` | s7/s13, L1035/L1041 |
|
||
| `%rd1` = `param_0+80` | `add.s64 %rd1, %rd8, 80` | s4, L1032 |
|
||
| **input plane arena** | `ld.param.u64 %rd34, [param_0+8]` → `cvta… %rd35, %rd34` | s29/s30, L1057–L1058 |
|
||
| **texture / skip input** | `ld.param.u64 %rd149, [%rd1+-56]` = `param_0+24` → `cvta… %rd4, %rd149` | s2215/s2216, L4641–L4642 |
|
||
| **output plane arena** | `ld.param.u64 %rd388, [%rd8+48]` → `cvta… %rd5, %rd388` | s15754/s15755, L38150/L38151 |
|
||
| block-origin table | `ld.param.v2.u16 {%rs296, %rs297}, [param_0+80]` | s5, L1033 |
|
||
| window index `tid.z` | `mov.u32 %r9, %tid.z` | s325, L1355 |
|
||
| shared arena | `.shared .align 4 .b8 _ZZ33…dec1_kernel…E4smem[6400]` | L1027 (declaration; not a statement) |
|
||
| per-lane weight address | `mul.wide.u32 %rd, %laneid, 16` + `add.s64 %rd, W, %rd` | every load block, e.g. s7214–s7215, L12685–L12686 |
|
||
| per-lane *table* address | `shl.b32 %r, %laneid, 2` → `cvt.u64.u32` → `and.b64 %rd, %r, 12` → `add.s64 %rd, %rd3, %rd` = `W + ((laneid·4)&12)` | e.g. s6684–s6686, L10864–L10866 |
|
||
|
||
The last row matters: the `ld.global.v2.u16` / `ld.global.u32` table loads in an
|
||
epilogue read `W + ((laneid<<2)&12) + off`, i.e. **the same 16-byte slot is read
|
||
by the four lanes with equal `laneid&3`, and consecutive loads pick consecutive
|
||
4-byte words inside one 64-byte record** — so those tables are indexed by
|
||
column, not by row.
|
||
|
||
Two per-`tid.z` strides exist and are used by *different* groups:
|
||
|
||
| group | base | statements |
|
||
|---|---|---|
|
||
| phase 1's B | `W + 12288·tid.z + 16·laneid` | s326–s331, L1356–L1363 |
|
||
| phase 1's C bias | `W + 256·tid.z + ((laneid·4)&12) + 24576…24816` | s379–s402, L1459–L1484 |
|
||
| phases 2…5 B + phase 6's bias | `W + 18432·tid.z + 25216` (`%rd307`, s7073) | s7068–s7073, L11935–L11940 |
|
||
|
||
`%rd307` is written exactly once (s7073) and consumed only by s7092–s7095
|
||
(phase 2's B) and by the four load blocks of E2…E5 (s7122, s7153, s7184, s7215).
|
||
Every other phase's loads go straight off `W` with absolute offsets.
|
||
|
||
### 1.2 Constants that survive between epilogues
|
||
|
||
| register | first definition | f32/f64 literal | value used (after the `cvt` to f16) |
|
||
|---|---|---|---|
|
||
| `%f476` | s7433 `mov.f32 %f476, 0f3C8CCB50` | 0.017186790704727173 | f16 `0x2466` = 0.017181396484375 |
|
||
| `%fd383` | s7437 `mov.f64 %fd383, 0dBFE1CC0000000000` | −0.55615234375 | f16 `0xB873` |
|
||
| `%fd385` | s7442 `mov.f64 %fd385, 0d3FE1CC0000000000` | +0.55615234375 | f16 `0x3873` |
|
||
| `%f478` | s7448 `mov.f32 %f478, 0f3F6D6000` | 0.92724609375 | f16 `0x3B6B` |
|
||
| `%f480` | s7453 `mov.f32 %f480, 0f3FB00000` | 1.375 | f16 `0x3D80` |
|
||
| `%fd1` | s6143 `mov.f64 %fd1, 0d3F20000000000000` | 2⁻¹³ | f16 `0x0800` = 0.0001220703125 |
|
||
| `%f989` | s12414 `mov.f32 %f989, 0f3ED306EB` | 0.4121621549129486 | f16 `0x3698` = 0.412109375 |
|
||
| `%f990` | s12417 `mov.f32 %f990, 0f3DA60DD6` | 0.0810810774564743 | f16 `0x2D30` = 0.0810546875 |
|
||
| `%f991` | s12420 `mov.f32 %f991, 0f3F000000` | 0.5 | f16 `0x3800` |
|
||
| `%f992` | s12423 `mov.f32 %f992, 0f40000000` | 2.0 | f16 `0x4000` |
|
||
|
||
`%f476/%fd383/%fd385/%f478/%f480` are defined **inside E6** and reused by the
|
||
second softmax — there is no second softmax here, so they are consumed by E6
|
||
alone. `%f989…%f992` are defined **inside E10** (s12414–s12425) and re-read by
|
||
E12, E14, E16, E18, E20, E22, E24 (e.g. s12642 `cvt.rn.f16.f32 low, %f989`);
|
||
E12 is the only one that materialises them; the others only `cvt` them.
|
||
|
||
Also live across the region (defined in the prologue, read by every warp
|
||
reduction): `%r5735 = 1` (s5579, L8209), `%r5744 = 2` (s6061, L9582),
|
||
`%r5746 = -1` (s6054, L9566) — the three operands of every
|
||
`shfl.sync.bfly.b32` in the epilogues. (`%r5746` is also reused as the
|
||
membermask of the E6 shuffles.)
|
||
|
||
### 1.3 Two quantisation helpers
|
||
|
||
* `cvt.rn.satfinite.e4m3x2.f16x2` — the requantisation used everywhere
|
||
(`f16x2_to_e4m3x2_satfinite_bits` in `ptx/lib/zluda_ptx_impl.cpp`).
|
||
* `cvt.rn.f16x2.e4m3x2` — its inverse, the *unpack*, used only when a shared or
|
||
register tile that was stored e4m3-packed is read back (E1: 144 of them,
|
||
s5590–s5676, L8241–L9556).
|
||
* `rcp.approx.ftz.f32`, `rsqrt.approx.ftz.f32`, `ex2.approx.ftz.f32`,
|
||
`div.approx.ftz.f32` — always wrapped as
|
||
`cvt.f32.f16 → op → cvt.rn.f16.f32` (round each step through f16).
|
||
|
||
## 2. Summary
|
||
|
||
| epilogue | range | stmts | named operation |
|
||
|---|---|---|---|
|
||
| E1 | s547–s7095 | 6549 | patch expand into shared + per-column RMS norm + gain + restage |
|
||
| E2 | s7120–s7126 | 7 | next-B weight loads (P3) |
|
||
| E3 | s7151–s7157 | 7 | next-B weight loads (P4) |
|
||
| E4 | s7182–s7188 | 7 | next-B weight loads (P5) |
|
||
| E5 | s7213–s7383 | 171 | score-bias C-seed loads (24×512 B) + 80 packs of P2/P3's D |
|
||
| E6 | s7432–s11274 | 3843 | cubic-exponent softmax, V transpose, pack |
|
||
| E7 | s11323–s11505 | 183 | pack P7's D → shared, restage A, next-B, per-column table + residual → P8's C |
|
||
| E8 | s11522–s11537 | 16 | 2 shared A fragments + 4 next-B loads (P9) |
|
||
| E9 | s11554–s12396 | 843 | RMS norm (squares, butterfly, +2⁻¹³, rsqrt, gain), packs, next B/C |
|
||
| E10 | s12413–s12833 | 421 | clamped-cubic activation + per-column table + residual, packs, next B |
|
||
| E11 | s12850–s12917 | 68 | 32 packs → P12's A, next B/C |
|
||
| E12 | s12934–s13241 | 308 | clamped-cubic activation, packs, next B |
|
||
| E13 | s13258–s13325 | 68 | 32 packs → P14's A, next B/C |
|
||
| E14 | s13342–s13649 | 308 | clamped-cubic activation, packs, next B |
|
||
| E15 | s13666–s13733 | 68 | 32 packs → P16's A, next B/C |
|
||
| E16 | s13750–s14057 | 308 | clamped-cubic activation, packs, next B |
|
||
| E17 | s14074–s14141 | 68 | 32 packs → P18's A, next B/C |
|
||
| E18 | s14158–s14465 | 308 | clamped-cubic activation, packs, next B |
|
||
| E19 | s14482–s14549 | 68 | 32 packs → P20's A, next B/C |
|
||
| E20 | s14566–s14873 | 308 | clamped-cubic activation, packs, next B |
|
||
| E21 | s14890–s14957 | 68 | 32 packs → P22's A, next B/C |
|
||
| E22 | s14974–s15281 | 308 | clamped-cubic activation, packs, next B |
|
||
| E23 | s15298–s15365 | 68 | 32 packs → P24's A, next B/C |
|
||
| E24 | s15382–s15689 | 308 | clamped-cubic activation, packs, next B |
|
||
| E25 | s15706–s15980 | 275 | pack → 16 guarded plane-arena scatter → B/C for P26 |
|
||
| E26 | s15989–s16304 | 316 | 6 D stores → shared, row max, softmax, 9+2 texture reads, surface store |
|
||
|
||
---
|
||
|
||
## 3. The epilogues
|
||
|
||
### E1 — s547–s7095: patch expand, RMS norm, gain, restage
|
||
|
||
6549 statements; op histogram (top): 1102 `add.s32`, 523 `shl.b32`, 460
|
||
`mov.b32`, 438 `$label`, 414 `sub.s32`, 368 `min.u32`, 341 `bra.uni`, 288
|
||
`mul.f16`, 198 `add.f16`, 144 `cvt.rn.satfinite.e4m3x2.f16x2`, 96
|
||
`cvt.rn.f16x2.e4m3x2`, 96 `rsqrt.approx.ftz.f32`, 96 `cvt.f32.f16`, 96
|
||
`cvt.rn.f16.f32`, 54 `add.f16x2`, 54 `st.shared.*` (46 `u32` + 2 `v2.u16` + 6
|
||
`v4.u32`), 36 `ld.shared.*` (24 `v2.u16` + 12 `v4.u32`), 32
|
||
`ld.global.v2.u16`, 12 `shfl.sync.bfly.b32`, 1 `ld.param`, 2 `ld.weak…`.
|
||
|
||
Sub-sections (bounded by the first/last statement of each op):
|
||
|
||
| range | what |
|
||
|---|---|
|
||
| s547–s688 | 144 packs of phase 1's D → the 48 b32 `%r10…%r55` |
|
||
| s689–s1367 | per-lane index arithmetic + the 48 guarded patch stores (46 `st.shared.u32` s732–s1355 + 2 `st.shared.v2.u16` s718/s1367); 1st `bar.sync` at s1369 |
|
||
| s1413–s2214 | 24 `ld.shared.v2.u16` read back of the packed patch |
|
||
| s2215–s2216 | `param_0+24` → `%rd4`: **the patch-gather buffer** |
|
||
| s2371–s5577 | 24 clamped `ld.global.v2.u16` gathered from that buffer |
|
||
| s5578–s5674 | 96 `cvt.rn.f16x2.e4m3x2` unpacks of the read-back patch |
|
||
| s5677–s6287 | the sum-of-squares trees (198 `add.f16`, 54 `add.f16x2`) |
|
||
| s6143–s6144 | the 2⁻¹³ round (s6143 `mov.f64 %fd1`, s6144 `cvt.rn.f16.f64 %rs1082, %fd1`) |
|
||
| s6293–s6670 | 96 `rsqrt.approx.ftz.f32`, each `cvt.f32.f16 → rsqrt → cvt.rn.f16.f32` |
|
||
| s6668–s6686 | 12 statements: **gain loads** `W + ((laneid·4)&12) + {25088…25200}` (8 `ld.global.v2.u16`) |
|
||
| s5868–s6973 | the 288 `mul.f16` (two groups: `inv · gain` then `x · (inv·gain)`) |
|
||
| s6990–s7047 | 24 packs → the 6 `st.shared.v4.u32` at s7008/7021/7034/7047/7061/7066 |
|
||
| s7058–s7089 | `bar.sync` (s7072) + 12 `ld.shared.v4.u32` (s7077–s7089) → **phase 2's A** |
|
||
| s7068–s7095 | `%rd307 = W + 18432·tid.z + 25216` (s7073) and 2 `ld.weak` → **phase 2's B** |
|
||
|
||
Evidence for the patch scatter (48 stores):
|
||
|
||
```
|
||
s718 L2859: st.shared.v2.u16 [%r2393], {%rs200, %rs201}
|
||
s732 L2875: st.shared.u32 [%r2400+1600], %r10
|
||
s771 L2916: st.shared.u32 [%r2418+1600], %r12
|
||
s783 L2932: st.shared.u32 [%r2425+3200], %r13
|
||
s797 L2948: st.shared.u32 [%r2432+4800], %r14
|
||
s7066 L11932: st.shared.v4.u32 [%r5949+4608], {%r5957, %r5956, %r5955, %r5954}
|
||
```
|
||
|
||
The 48 patch stores are 12 blocks of 4: base offsets `{0, 1600, 3200, 4800}`
|
||
(12 stores each) plus two `v2.u16` stores. Each block is guarded by
|
||
|
||
```
|
||
s708 L2848: setp.gt.u32 %p5, %r64, 9
|
||
s709 L2849: setp.gt.u32 %p6, %r62, 9
|
||
s711 L2851: @%p7 bra $L__BB1_2
|
||
```
|
||
|
||
and the index is `(row_clamped·40 + col_clamped·8 + t)·4` bytes
|
||
(s714 `mad.lo.s32 %r2389, %r64, 40, %r2388`, s715 `or.b32 %r2390, %r57, %r2389`,
|
||
s716 `shl.b32 %r2391, %r2390, 2`), i.e. a **10×10 grid of 4-word cells × 4
|
||
sub-arrays = 6400 bytes = the whole shared allocation**. (§7/U4: the guards
|
||
bound row/col at 9; I did not reduce the 438-label duplication to a closed form.)
|
||
|
||
The gather from `param_0+24` (s2215–s2371) uses
|
||
|
||
```
|
||
s2245 L4674: mad.lo.s32 %r3019, %r3009, %r7, %r3018 // clamp_row·(8·ex) + clamp_col·8
|
||
s2250 L4679: add.s32 %r19034, %r3023, %r3019 // + clamp(t, - , 7)
|
||
s2369 L4806: mul.wide.u32 %rd150, %r19034, 4
|
||
s2371 L4808: ld.global.v2.u16 {%rs346, %rs347}, [%rd151]
|
||
```
|
||
|
||
so the element stride of the `+24` buffer is 4 bytes (one f16x2 = two channels),
|
||
indexed `row·(8·ex) + col·8 + clamp(t,0,7)`; the same buffer is read 24 times
|
||
per lane, once per `(row,col)` of the stencil.
|
||
|
||
The norm itself:
|
||
|
||
```
|
||
s6143 L9797: mov.f64 %fd1, 0d3F20000000000000 // 2^-13
|
||
s6144 L9799: cvt.rn.f16.f64 %rs1082, %fd1
|
||
s6186 L9927: add.f16 %rs1161,%rs1067,%rs1082 // ...+ eps
|
||
s6293 L10243: rsqrt.approx.ftz.f32 fl, fl
|
||
s6688 L10878: mul.f16 %rs1371,%rs1372,%rs2011 // inv * gain
|
||
s6832 L11313: mul.f16 %rs1659,%rs488,%rs1371 // value * (inv*gain)
|
||
```
|
||
|
||
**There is no mean subtraction and no `1/N`.** The accumulated quantity is the
|
||
raw sum of squares of the patch's 3 channels plus the constant 2⁻¹³, and the
|
||
norm is applied per *column* (the gain is read at `W + ((laneid<<2)&12) + off`
|
||
so it is constant across rows). The 12 `shfl.sync.bfly.b32` sum across the four
|
||
lanes of each `laneid&3` group.
|
||
|
||
**Inputs:** phase 1's D (`%r692…%r2083`), `W`, `param_0+8`, `param_0+24`,
|
||
`%r1/%r2/%r7/%r8/%r2195/%r2196`, `%r9` (`tid.z`).
|
||
**Outputs (live past s7095):** the 6 patch-expand `st.shared` regions, the 12
|
||
`ld.shared.v4.u32` A fragments `%r18458…%r18505`, phase 2's B
|
||
`%r5960…%r5967`, `%rd307`, the 5 constants of §1.2.
|
||
|
||
### E2/E3/E4 — s7120–s7126, s7151–s7157, s7182–s7188: next-B loads
|
||
|
||
Three 7-statement blocks, each identical in shape:
|
||
|
||
```
|
||
s7120 L12140: mov.u32 %r6208, %laneid
|
||
s7121 L12142: mul.wide.u32 %rd309, %r6208, 16
|
||
s7122 L12143: add.s64 %rd310, %rd307, %rd309
|
||
s7123 L12144: add.s64 %rd203, %rd310, 2048
|
||
s7124 L12146: ld.weak.global.ca.v4.u32 { %r6209,…%r6212},[%rd203] // +2560 as well
|
||
```
|
||
|
||
| epilogue | offsets added to `%rd307` (`W + 18432·tid.z + 25216`) | destination phase |
|
||
|---|---|---|
|
||
| E2 | `+2048`, `+2560` | P3's B |
|
||
| E3 | `+1024`, `+1536` | P4's B |
|
||
| E4 | `+3072`, `+3584` | P5's B |
|
||
|
||
Nothing else. The offsets are deliberately out of phase order: the weight
|
||
sub-image is laid out P2, P4, P3, P5 (offset blocks 0, 1, 2, 3 of 1024 bytes).
|
||
|
||
### E5 — s7213–s7383: score-bias C-seed loads + pack of P2/P3's D
|
||
|
||
171 statements; the only epilogue whose weight loads carry the `tid.z` stride.
|
||
|
||
* **24 loads** `W + 18432·tid.z + 16·laneid + 25216 + 4096 + 512·i`,
|
||
`i = 0…23`, at s7217…s7263 (L12689…L12781), into `%r6956…%r7051`. Byte block
|
||
`[18432·tid.z + 29312, +41100)` = 12288 bytes = 24 tiles × 512 — **phase 6's C
|
||
score-bias seed**.
|
||
* **80 packs / 40 b32**, s7264–s7383, whose sources are registers written by
|
||
**phase 2's and phase 3's mma** (`%r6466/6467/6486/6487/…` from s7096–s7126
|
||
and s7127–s7136). Two interleaved destinations — `%r7498 %r7508 …` and
|
||
`%r7244 %r7246 …` — which become phase 6's B and A respectively (verified:
|
||
`%r7244 ← mov.b32 {%rs2061,%rs2062}` at s7338 whose sources are
|
||
`cvt` of `%r6466/%r6486`; `%r7498 ← mov.b32 {%rs2013,%rs2014}` at s7266 whose
|
||
source is `cvt` of `%r6466` at s7265).
|
||
|
||
**The C-seed layout.** The four C registers of one mma are the four u32 of one
|
||
16-byte lane slot, and the tile index runs `4·(n>>1) + m` — the same scheme
|
||
`rrswin_enc0_epilogues.md` §3/E9 established. Checked against phase 6's own
|
||
operands: `s7384 C=%r6956,%r6957`, `s7385 C=%r6958,%r6959`,
|
||
`s7390 C=%r7004,%r7005`, `s7391 C=%r7006,%r7007` — consecutive pairs from the
|
||
load block, i.e. mma `(m, n)` takes the slot `(n>>1)` of the load block.
|
||
|
||
### E6 — s7432–s11274: cubic-exponent softmax, V transpose, pack
|
||
|
||
**Identical in kind to `cuda_dldn_engine_swin_enc0_kernel`'s E5** (see
|
||
`rrswin_enc0_epilogues.md` §3/E5) and byte-for-byte the same op histogram:
|
||
1048 `mov.b32`, 672 `cvt.rn.f16.f32`, 384 `cvt.rn.f16.f64`, 192 each of
|
||
`fma.rn.f16x2` / `cvt.f32.f16` / `rcp.approx.ftz.f32` / `mul.f16`, 144 packs,
|
||
48 `movmatrix`, 16 `shfl.sync.bfly.b32`, 96 each of `max/min/neg/abs.f16x2`.
|
||
|
||
Sub-sections:
|
||
|
||
| range | stmts | what |
|
||
|---|---|---|
|
||
| s7433–s7458 | 26 | the 5 constants (§1.2): s7433 `mov.f32 %f476, 0f3C8CCB50`, s7437/s7442 the two f64 clamps, s7448/s7453 the cubic pair |
|
||
| s7459–s9549 | 2091 | 96 per-element groups (scale, clamp, cubic, `and.b32 …,2145419232`) |
|
||
| s9550–s9635 | 86 | in-lane partial sums (`add.f16x2`) |
|
||
| s9636–s9765 | 130 | 16 `shfl.sync.bfly` → 8 row sums |
|
||
| s9766–s10587 | 822 | 192 `rcp.approx.ftz.f32` → 96 reciprocals |
|
||
| s10588–s11010 | 423 | 192 `mul.f16` → 96 probabilities |
|
||
| s11011–s11058 | 48 | `movmatrix.sync.trans.aligned.m8n8.b16` of **phase 5's** D |
|
||
| s11059–s11273 | 215 | 144 `cvt.rn.satfinite.e4m3x2.f16x2` packs |
|
||
|
||
Evidence:
|
||
|
||
```
|
||
s7459 L13474: and.b32 %r9534, %r18506, 2145419232 // 0x7FE07FE0
|
||
s9636 L20270: shfl.sync.bfly.b32 %r9778,%r9774,%r5735,%r9781,%r5746
|
||
s9766 L20596: rcp.approx.ftz.f32 fl, fl
|
||
s11011 L22895: movmatrix.sync.trans.aligned.m8n8.b16 %r10138, %r6715
|
||
s11059 L23039: cvt.rn.satfinite.e4m3x2.f16x2 %rs3462, %r10146
|
||
```
|
||
|
||
The transposed registers `%r6715…%r6946` are exactly phase 5's D (s7189–s7212).
|
||
`%r10138/%r10146/…` are the movmatrix results; the pack at s11059 takes
|
||
`%r10146`, i.e. the transposed V is what becomes **phase 7's B** (checked:
|
||
`%r10744 ← mov.b32 {%rs3461,%rs3462}` at s11061, and `%r10744` is phase 7's
|
||
first B pair; the probability packs, e.g. `%r10450 ← mov.b32
|
||
{%rs3509,%rs3510}` at s11133, feed phase 7's A). `%r6715` is phase 5's first D
|
||
and `%r6946` its last, so all 48 movmatrix inputs are phase 5's D.
|
||
|
||
For the record, the per-element function (identical to enc0's E5(a), reproduced
|
||
here because it is the only arithmetic in this file that is *not* obviously a
|
||
named primitive):
|
||
|
||
```
|
||
m = clamp(f16(x * 0.017181396484375), -0.55615234375, +0.55615234375)
|
||
u = f16(1.375 + m*(0.92724609375 - m*m))
|
||
expval = f16 from bits ((u_bits << 5) & 0x7FE07FE0)
|
||
```
|
||
|
||
**There is no row maximum in E6** (the only `max.f16x2`/`min.f16x2` are the 96
|
||
clamp pairs of (a)), and no bias add — the bias is phase 6's GEMM C operand.
|
||
|
||
**Inputs:** phase 6's 96 D f16x2, phase 5's 48 D f16x2, `%r5735/%r5744/%r5746`.
|
||
**Outputs:** the 5 constants, phase 7's 12 A fragments + 12 B fragments.
|
||
**Memory:** none (0 `ld.*`, 0 `st.*` in 3843 statements).
|
||
|
||
### E7 — s11323–s11505: pack P7's D → shared, restage, next-B, table → P8's C
|
||
|
||
183 statements: 64 `add.f16`, 48 `mov.b32`, 32 packs, 8 `ld.global.v2.u16`,
|
||
4 `st.shared.v4.u32`, 4 `ld.shared.v4.u32`, 4 `ld.weak`, 6 `add.s64`.
|
||
|
||
```
|
||
s11323 L23878: bar.sync 0
|
||
s11339 L23912: st.shared.v4.u32 [%r18635], {%r18639, %r18638, %r18637, %r18636}
|
||
s11352 L23941: st.shared.v4.u32 [%r18635+512], {…}
|
||
s11365 L23970: st.shared.v4.u32 [%r18635+1024], {…}
|
||
s11378 L23999: st.shared.v4.u32 [%r18635+1536], {…}
|
||
s11379 L24000: bar.sync 0
|
||
s11384 L24007: ld.shared.v4.u32 {%r18655,…%r18658}, [%r18654]
|
||
s11385 L24008: ld.shared.v4.u32 {%r18659,…%r18662}, [%r18654+512]
|
||
s11390 L24016: ld.weak.global.ca.v4.u32 { %r10893,…%r10896},[%rd233] // W+41600
|
||
s11402 L24037: ld.global.v2.u16 {%rs5218, %rs5219}, [%rd322+62096]
|
||
s11410 L24046: add.f16 %rs3640,%rs5233,%rs491
|
||
```
|
||
|
||
* **32 packs of phase 7's D** (`%r10378…%r10428`, the 192×96 banded P·V output)
|
||
→ 16 b32 → four `st.shared.v4.u32` at `[%r18635 + {0,512,1024,1536}]`; the
|
||
`bar.sync` pair makes them visible, then **two `ld.shared.v4.u32` give phase
|
||
8's A** (`%r18655…%r18662`).
|
||
* **4 `ld.weak` at `W + 16·laneid + {41600, 42112, 42624, 43136}`** → phase 8's B.
|
||
* **8 `ld.global.v2.u16` at `W + ((laneid·4)&12) + {62080…62192}`** — a
|
||
per-column vector, 8 words per lane = 32 columns.
|
||
* **64 `add.f16`** adding that vector to the *f16 halves of phase 7's D* which
|
||
were unpacked at s6186–s6288 (`%rs488/%rs491/%rs494…`), producing
|
||
`%r10918 %r10919 %r10928 %r10929 …` — **phase 8's C operand** (confirmed:
|
||
`s11506 C=%r10918,%r10919`, `s11507 C=%r10928,%r10929`). So phase 8 computes
|
||
`D8 = A8·B8 + (D7 + column_table)`.
|
||
|
||
### E8 — s11522–s11537: next-A and next-B for P9
|
||
|
||
```
|
||
s11525 L24450: ld.shared.v4.u32 {%r18666,…%r18669}, [%r18665+2048]
|
||
s11526 L24451: ld.shared.v4.u32 {%r18670,…%r18673}, [%r18665+2560]
|
||
s11531 L24459: ld.weak.global.ca.v4.u32 { %r11072,…%r11075},[%rd237] // W+60032
|
||
s11537 L24471: ld.weak.global.ca.v4.u32 { %r11084,…%r11087},[%rd240] // W+61568
|
||
```
|
||
|
||
Two shared A fragments (the upper half of the same staging region, offsets 2048
|
||
and 2560 relative to `%r18652`) and four weight fragments at
|
||
`W + {60032, 60544, 61056, 61568}` → phase 9's A and B.
|
||
|
||
### E9 — s11554–s12396: RMS norm before the second attention window
|
||
|
||
843 statements, no stores, 16 memory ops. Structure:
|
||
|
||
| range | stmts | op |
|
||
|---|---|---|
|
||
| s11554–s11699 | 146 | 32 × `x·x` squares of phase 9's D |
|
||
| s11700–s11757 | 58 | in-lane pairwise `add.f16x2` |
|
||
| s11714–s11757 | 44 | 8 `shfl.sync.bfly` (2 per row) → row sums |
|
||
| s11764–s11779 | 16 | `add.f16` of the two halves, then `+ 2⁻¹³` (§1.2 `%fd1`) |
|
||
| s11779–s11880 | 102 | broadcast each row sum into 4 b32 |
|
||
| s11883–s12128 | 246 | 64 `rsqrt.approx.ftz.f32` |
|
||
| s12129–s12136 | 8 | **gain loads** `W + ((laneid·4)&12) + {62208…62320}` |
|
||
| s12137–s12232 | 96 | `mul.f16` — `inv · gain` |
|
||
| s12233–s12328 | 96 | `mul.f16` — `x · (inv·gain)` → 32 b32 `%r17567…%r17598` |
|
||
| s12332–s12339 | 8 | next-B loads `W + {62336, 62848, 63360, 63872}` → phase 10's B |
|
||
| s12341–s12348 | 8 | next-C loads `W + ((laneid·4)&12) + {64384,64400,64416,64432}` → phase 10's C |
|
||
| s12349–s12396 | 48 | 32 packs → phase 10's A (`%r11574…%r11577`, `%r11594…`, …) |
|
||
|
||
Evidence:
|
||
|
||
```
|
||
s11557 L24589: mul.f16 %rs3832,%rs4647,%rs4647 // square of a D half
|
||
s11714 L25018: shfl.sync.bfly.b32 %r11325,%r11321,%r5735,%r11328,%r5746
|
||
s11776 L25176: cvt.rn.f16.f64 %rs4033, %fd1 // eps = 2^-13
|
||
s11883 L25588: rsqrt.approx.ftz.f32 fl, fl
|
||
s12129 L25883: ld.global.v2.u16 {%rs5234, %rs5235}, [%rd327+62224]
|
||
s12233 L26180: mul.f16 %rs4421,%rs4647,%rs4229 // D_half * (inv*gain)
|
||
s12332 L26472: add.s64 %rd241, %rd329, 62336
|
||
s12349 L26500: cvt.rn.satfinite.e4m3x2.f16x2 %rs4611, %r17568
|
||
```
|
||
|
||
**What it computes.** For each of the 8 rows a lane holds:
|
||
|
||
```
|
||
sumsq[row] = Σ_c x[row,c]² (f16, 4-lane butterfly)
|
||
inv = rsqrt.approx.ftz.f32(f32(f16(sumsq + 2⁻¹³))) (rounded to f16)
|
||
out[row,c] = x[row,c] · inv · gain[c]
|
||
```
|
||
|
||
* **no mean subtraction and no `1/N`** — the quantity is the raw sum of squares
|
||
plus the constant 2⁻¹³.
|
||
* `gain[c]` is per *column* (the `W + ((laneid<<2)&12)` pattern) and constant
|
||
across rows.
|
||
* the `inv` register is duplicated into both halves of a b32 before use, so one
|
||
row-sum serves every n-tile of that row (64 `rsqrt` for 16 row sums).
|
||
* the norm's *bias* is not added here; it is phase 10's C, loaded at the end of
|
||
this epilogue (s12345–s12348).
|
||
|
||
**Outputs:** `%r11473 %r11477` (P10 B), `%r11611 %r11621 %r11651 %r11661`
|
||
(P10 C), 16 packed A registers, the 32 f16x2 normed values `%r17567…%r17598`
|
||
(re-packed by E11).
|
||
|
||
#### Shape of the attention region (E5–E9)
|
||
|
||
```
|
||
P2 A = patch(shared), B = W+25216+2048, C = 0 → D2 (96×32)
|
||
P3 A = patch(shared), B = W+25216+ 0, C = 0 → D3 (96×32)
|
||
P4 A = patch(shared), B = W+25216+1024, C = D2 → D4
|
||
P5 A = patch(shared), B = W+25216+3072, C = D3 → D5
|
||
E5 packs D2 ∪ D3 → P6 A and B; E5 loads P6 C = bias@29312
|
||
P6 A = pack(D2∪D3), B = pack(D2∪D3), C = bias → scores 64×96
|
||
E6: softmax over the 96 columns (no row max), 48 movmatrix(D5), pack
|
||
P7 A = pack(prob), B = pack(transpose(D5)), C = 0 → D7 (192×96)
|
||
E7: pack D7 → shared; P8 C = D7 + column table@62080; P8 B = W+41600
|
||
P8 A = shared, B = W+41600, C = D7+table → D8
|
||
P9 A = shared+2048, B = W+60032, C = 0 → D9
|
||
E9: RMS norm of D9 + gain, packs → P10 A
|
||
```
|
||
|
||
### E10 — s12413–s12833: clamped-cubic activation, per-column table, residual
|
||
|
||
421 statements; this is the only epilogue that materialises `%f989…%f992`.
|
||
|
||
```
|
||
s12414 L26726: mov.f32 %f989, 0f3ED306EB
|
||
s12420 L26740: mov.f32 %f991, 0f3F000000 // 0.5
|
||
s12426 L26755: neg.f16x2 %r11687,%r11686
|
||
s12427 L26759: max.f16x2 %r11689,%r11542,%r11687
|
||
s12428 L26763: min.f16x2 %r11692,%r11689,%r11686
|
||
s12429 L26767: abs.f16x2 %r11695,%r11692
|
||
s12432 L26779: mul.f16x2 %r11703,%r11692,%r11700
|
||
s12433 L26783: add.f16x2 %r11706,%r11685,%r11703
|
||
s12434 L26787: mul.f16x2 %r11709,%r11542,%r11706
|
||
```
|
||
|
||
Per element, over each of the 16 f16x2 D registers of phase 10:
|
||
|
||
```
|
||
y = clamp(x, -2, +2)
|
||
g = 0.5 + y * (0.412109375 - 0.0810546875 * |y|) // f16 throughout
|
||
out = x * g
|
||
```
|
||
|
||
`g` is a *cubic ramp*, not a sigmoid: `g(+2)=1 ⇒ out=x` for `x ≥ 2`;
|
||
`g(-2)=0 ⇒ out=0` for `x ≤ -2`. **Reproduce the formula, not SiLU/GELU.**
|
||
|
||
Then:
|
||
|
||
| range | op |
|
||
|---|---|
|
||
| s12413–s12689 | 16 activation groups |
|
||
| s12693–s12700 | next-B loads `W + {64448,64960,65472,65984}` → phase 11's B |
|
||
| s12701–s12713 | `ld.global.v2.u16` at `W + ((laneid·4)&12) + {66496…66608}` |
|
||
| s12714–s12809 | 64 `add.f16` — residual add, written over the **activation inputs** (`%rs4644/%rs4647`, unpacked at s11556/… from phase 9's D) |
|
||
| s12810–s12833 | 16 packs of the activation outputs → phase 12's A |
|
||
|
||
Evidence for the residual add:
|
||
|
||
```
|
||
s12706 L27718: ld.global.v2.u16 {%rs5250, %rs5251}, [%rd337+66512]
|
||
s12714 L27727: add.f16 %rs4645,%rs5265,%rs4647
|
||
s12716 L27734: mov.b32 %r12189, {%rs4642, %rs4645}
|
||
s12810 L28015: cvt.rn.satfinite.e4m3x2.f16x2 %rs4835, %r11825
|
||
```
|
||
|
||
The residual is **phase 9's D**, not phase 10's: `%rs4644/%rs4647` are the halves
|
||
of `%r11088`, unpacked at s11556 (`mov.b32 {%rs4644, %rs4647}, %r11088`) at the
|
||
very start of E9, and `%r11088` is written by phase 9's mma (s11538–s11553).
|
||
The 32 such registers `%r12189 %r12190 … %r12329 %r12330` are **phase 11's C**;
|
||
`%r12148 %r12152 …` are phase 11's B.
|
||
|
||
Note the two pack streams: the *activation* outputs (`%r11709`, `%r11825`, …) go
|
||
to phase 12's A, while the *residual+table* registers go to phase 11's C.
|
||
|
||
### E11 — s12850–s12917: packs of the normed tile, next B/C
|
||
|
||
68 statements: 32 packs, 4 `ld.weak`, 4 `ld.global.u32`.
|
||
|
||
```
|
||
s12854 L28189: ld.weak.global.ca.v4.u32 { %r12342,…%r12345},[%rd249]
|
||
s12866 L28210: ld.global.u32 %r12480, [%rd342+68672]
|
||
s12870 L28216: cvt.rn.satfinite.e4m3x2.f16x2 %rs…, %r17567
|
||
```
|
||
|
||
* four `v4.u32` fragments at `W + {66624, 67136, 67648, 68160}` → phase 12's B;
|
||
* four `ld.global.u32` at `W + ((laneid·4)&12) + {68672, 68688, 68704, 68720}`
|
||
→ **phase 12's C seed**, each used in both halves of the C operand;
|
||
* **32 packs of `%r17567…%r17598`** — the *normalised* tile E9 produced — so
|
||
phase 12's A operand is bit-identical to phase 10's A. Phases 10 and 12 are
|
||
two parallel GEMMs off the same normed tile.
|
||
|
||
### E12 — s12934–s13241: clamped-cubic activation, no residual
|
||
|
||
308 statements; op mix 72 `mov.b32`, 64 `cvt.rn.f16.f32`, 48 `mul.f16x2`, 16
|
||
each `neg/max/min/abs/sub/add.f16x2`, 4 `ld.weak`, 16 packs. **No constants of
|
||
its own** (it re-`cvt`s `%f989` etc. at s12642-equivalent sites) and **no
|
||
`ld.global`**.
|
||
|
||
Same function as E10; 16 packs of the activation outputs
|
||
(`%r13046` at s13191, …) → phase 13's A, plus 4 weight loads at
|
||
`W + {68736, 69248, 69760, 70272}` → phase 13's B. **No table add and no
|
||
residual**, so phase 13's C is zero.
|
||
|
||
### E13 — s13258–s13325: packs of the normed tile again, next B/C
|
||
|
||
68 statements, shape-identical to E11:
|
||
|
||
```
|
||
s13262 L29597: ld.weak.global.ca.v4.u32 { %r13210,…},[%rd257]
|
||
s13274 L29618: ld.global.u32 %r13348, [%rd349+72832]
|
||
```
|
||
|
||
B at `W + {70784, 71296, 71808, 72320}` → phase 14's B; **32 packs of
|
||
`%r17567…%r17598` — the normed tile for the second time**; C at
|
||
`W + ((laneid·4)&12) + {72832, 72848, 72864, 72880}` → phase 14's C.
|
||
|
||
### E14/E16/E18/E20/E22/E24 — s13342–s13649, s13750–s14057, s14158–s14465, s14566–s14873, s14974–s15281, s15382–s15689: activation + packs + next B
|
||
|
||
Six shape-identical 308-statement epilogues (shape hash `4234ec4de215`). Each
|
||
applies the E10 activation to the D of its own phase, then 16 packs of the
|
||
activation outputs → the next odd phase's A, and 4 weight loads → the next odd
|
||
phase's B. Evidence (E14):
|
||
|
||
```
|
||
s13376 L30936: cvt.rn.f16.f32 low, %f989
|
||
s13393 L30985: mul.f16x2 %r13727,%r13634,%r13724
|
||
s13619 L30816: ld.weak.global.ca.v4.u32 { %r13885,…},[%rd261] // W+70784
|
||
s13633 L30986: cvt.rn.satfinite.e4m3x2.f16x2 %rs…, %r13727
|
||
```
|
||
|
||
Offsets: E12→{68736,69248,69760,70272}, E14→{70784,71296,71808,72320},
|
||
E16→{72896,73408,73920,74432}, E18→{74944,75456,75968,76480},
|
||
E20→{77056,77568,78080,78592}, E21's targets … (from the phase table's weight
|
||
column: P13=68736, P15=72896, P17=77056, P19=81216, P21=85376, P23=89536,
|
||
P25=93696).
|
||
|
||
No `ld.global`, no table add, no residual: these six produce only A and B.
|
||
|
||
### E15/E17/E19/E21/E23 — s13666–s13733, s14074–s14141, s14482–s14549, s14890–s14957, s15298–s15365: packs + next B/C
|
||
|
||
Five shape-identical 68-statement epilogues (shape hash `819f37d58cd2`), the
|
||
twin of E11/E13. Example (E15):
|
||
|
||
```
|
||
s13670 L31005: ld.weak.global.ca.v4.u32 { %r14078,…},[%rd265]
|
||
s13682 L31026: ld.global.u32 %r14216, [%rd356+76992]
|
||
s13686 L31031: cvt.rn.satfinite.e4m3x2.f16x2 %rs…, %r17567
|
||
```
|
||
|
||
B at `W + {72896+1024…}` etc. per the phase table; C at
|
||
`W + ((laneid·4)&12) + {76992, 77008, 77024, 77040}` (E15 → phase 16's C);
|
||
**32 packs of `%r17567…%r17598`, the normed tile — third, fourth, fifth, sixth
|
||
and seventh time** (E13, E15, E17, E19, E21, E23 all pack the same 32 f16x2).
|
||
|
||
#### Shape of the whole MLP region (E9–E25)
|
||
|
||
```
|
||
E9 normed tile N (32 f16x2, %r17567…%r17598, s12349 packs)
|
||
├─ pack → P10 A (s12349) P10: A=N, B=W10, C=bias@64384 → act (E10)
|
||
│ pack of act → P12 A (s12870)
|
||
│ E10: P11 C = D9 + table@66496
|
||
├─ pack → P12 A (s12870) P12: A=N, B=W12, C=bias@68672 → act (E12)
|
||
│ pack of act → P14 A (s13394)
|
||
├─ pack → P14 A (s13394) P14: A=N, B=W14, C=bias@72832 → act (E14)
|
||
├─ pack → P16 A (s13806) P16: A=N, B=W16, C=bias@76992 → act (E16)
|
||
├─ pack → P18 A (s14218) P18: A=N, B=W18, C=bias@81152 → act (E18)
|
||
├─ pack → P20 A (s14630) P20: A=N, B=W20, C=bias@85312 → act (E20)
|
||
├─ pack → P22 A (s15042) P22: A=N, B=W22, C=bias@89472 → act (E22)
|
||
└─ pack → P24 A (s15454) P24: A=N, B=W24, C=bias@93632 → act (E24)
|
||
pack of act → P25 A (s15666)
|
||
```
|
||
|
||
The odd phases P11, P13, …, P25 take `C = 0` and produce the residual pair that
|
||
the following even phase's table adds.
|
||
|
||
### E25 — s15706–s15980: pack → 16 guarded plane-arena stores → B/C for P26
|
||
|
||
275 statements; the only `st.global` in the entry (16 of them, all here).
|
||
|
||
**(a) requantise, s15706–s15753.** 24 groups of `2 cvt + 1 mov.b32` → 24 b32;
|
||
the 16 consumed are `%r489…%r504`.
|
||
|
||
**(b) plane-arena scatter, s15754–s15960.**
|
||
|
||
```
|
||
s15754 L38150: ld.param.u64 %rd388, [%rd8+48] // = param_0+48
|
||
s15755 L38151: cvta.to.global.u64 %rd5, %rd388
|
||
s15757 L38155: shr.u32 %r505, %r18449, 2 // g = laneid>>2
|
||
s15758 L38156: and.b32 %r506, %r18449, 3 // t = laneid&3
|
||
s15761 L38159: add.s32 %r18707, %r488, %r18705 // y = %r488 + (laneid>>5)
|
||
s15767 L38165: add.s32 %r18713, %r1, %r18706 // x = %r1 + (g&7)
|
||
s15772 L38170: shl.b32 %r18714, %r18713, 3 // x<<3
|
||
s15773 L38171: mad.lo.s32 %r507, %r7, %r18707, %r18714
|
||
s15775 L38173: selp.b32 %r508, -1, %r18715, %p1
|
||
s15780 L38179: st.global.u32 [%rd390], %r489
|
||
```
|
||
|
||
Stride register `%r7 = 8·ex` (s15). Every store is
|
||
`idx = 8·ex·y + 8·x + d`, with `d = t` or `t|4`; each of the four base slices is
|
||
guarded by its own predicate of the form
|
||
`(y<0)|(y≥ey)|(laneid>127)|(x<0)|(x≥ex)` (e.g. s15762–s15771,
|
||
L38160–L38169); a guard failure stores `-1` as the index, which the following
|
||
`setp.lt.s32 %p, %idx, 0` + `bra` turns into "skip the store"
|
||
(s15774–s15777).
|
||
|
||
Store values, bases and column offsets (`g = laneid>>2`, `t = laneid&3`,
|
||
`%r488 = %r2 + 4·tid.z` at s11555/L24586, `%r8 = ey·8·ex`):
|
||
|
||
| store | stmt | value | index base | d |
|
||
|---|---|---|---|---|
|
||
| 1 | s15780 | `%r489` | `idx1` | `t` |
|
||
| 2 | s15789 | `%r490` | `idx1` | `t\|4` |
|
||
| 3 | s15815 | `%r491` | `idx2` | `t` |
|
||
| 4 | s15823 | `%r492` | `idx2` | `t\|4` |
|
||
| 5 | s15832 | `%r493` | `idx1 + %r8` | `t` |
|
||
| 6 | s15840 | `%r494` | `idx1 + %r8` | `t\|4` |
|
||
| 7 | s15849 | `%r495` | `idx2 + %r8` | `t` |
|
||
| 8 | s15857 | `%r496` | `idx2 + %r8` | `t\|4` |
|
||
| 9 | s15883 | `%r497` | `idx3` | `t` |
|
||
| 10 | s15891 | `%r498` | `idx3` | `t\|4` |
|
||
| 11 | s15917 | `%r499` | `idx4` | `t` |
|
||
| 12 | s15925 | `%r500` | `idx4` | `t\|4` |
|
||
| 13 | s15934 | `%r501` | `idx3 + %r8` | `t` |
|
||
| 14 | s15942 | `%r502` | `idx3 + %r8` | `t\|4` |
|
||
| 15 | s15951 | `%r503` | `idx4 + %r8` | `t` |
|
||
| 16 | s15959 | `%r504` | `idx4 + %r8` | `t\|4` |
|
||
|
||
with the four bases built as `mad.lo.s32 %rBASE, 8·ex, y, 8·x`:
|
||
|
||
| base | stmt | y | x |
|
||
|---|---|---|---|
|
||
| `idx1 = %r507` | s15773, L38171 | `%r488 + (laneid>>5)` | `%r1 + (g&7)` |
|
||
| `idx2 = %r511` | s15808, L38210 | `%r488 + ((g+8)>>3)` | `%r1 + ((g+8)&7)` |
|
||
| `idx3 = %r520` | s15876, L38290 | `%r488 + ((g+16)>>3)` | `%r1 + ((g+16)&7)` |
|
||
| `idx4 = %r523` | s15910, L38328 | `%r488 + ((g+24)>>3)` | `%r1 + ((g+24)&7)` |
|
||
|
||
The `g+8j` slices come from `add.s32 %r18717, %r18986, 8` (s15793, L38195),
|
||
`add.s32 %r18734, %r18988, 16` (s15861, L38275) and
|
||
`add.s32 %r18747, %r18990, 24` (s15895, L38313); each base carries its own
|
||
guard predicate (`p1…p4`) of the form
|
||
`(y<0)|(y≥ey)|(slice>31)|(x<0)|(x≥ex)`.
|
||
|
||
**(c) next B/C, s15961–s15980.**
|
||
|
||
```
|
||
s15962 L38392: ld.param.u64 %rd456, [param_0+40]
|
||
s15969 L38401: add.s64 %rd424, %rd456, %rd423 // + 16*laneid
|
||
s15971 L38404: ld.weak.global.ca.v4.u32 { %r18765,…%r18768},[%rd421] // +95744
|
||
s15973 L38408: ld.weak.global.ca.v4.u32 { %r18769,…%r18772},[%rd422] // +96256
|
||
s15979 L38417: ld.global.u32 %r18823, [%rd428+96768]
|
||
s15980 L38418: ld.global.u32 %r18833, [%rd428+96784]
|
||
```
|
||
|
||
Note `%rd424` is built from `%rd456` (the *un*cvtad pointer) while `%rd455` is
|
||
cvtad; on this corpus both are the same address, and the offsets still read
|
||
`W + ….`
|
||
|
||
**Outputs:** `%r489…%r504` (re-read as P26's A, see s15981), `%r18765 %r18769`
|
||
(P26 B), `%r18823 %r18833` (P26 C), `%rd388` (`param_0+48`).
|
||
|
||
### E26 — s15989–s16304: the final epilogue
|
||
|
||
316 statements; it consumes the six D accumulators of phase 26
|
||
(`%r18794 %r18795 %r18804 %r18805 %r18834 %r18835 %r18844 %r18845` — the
|
||
`s15996`…`s15988` mma results) and writes the surface.
|
||
|
||
**(a) staging, s15989–s16050.** 12 guarded `st.shared.u32` of the phase-26 D
|
||
pairs into a shared scratch, indexed from `%r540 = laneid>>2`, `%r541 = laneid&3`
|
||
and `%tid.z`:
|
||
|
||
```
|
||
s16000 L38488: st.shared.u32 [%r542], %r18794
|
||
s16004 L38493: st.shared.u32 [%r542+16], %r18804
|
||
s16017 L38507: st.shared.u32 [%r544], %r18795
|
||
s16019 L38510: st.shared.u32 [%r544+16], %r18805
|
||
```
|
||
|
||
**(b) coordinates and bounds check, s16051–s16084.**
|
||
|
||
```
|
||
s16059 L38555: mov.u32 %r18897, %tid.x
|
||
s16060 L38556: add.s32 %r18898, %r19003, %r18897 // %r19003 = tid.z*32
|
||
s16065 L38561: sub.s32 %r547, %r18898, %r18902
|
||
s16067 L38563: add.s32 %r549, %r1, %r547 // x
|
||
s16068 L38564: add.s32 %r550, %r19004, %r548 // y, %r19004 = ctaid.y*8 - origin
|
||
s16073 L38569: ld.param.v2.u32 {%r18912, %r18913}, [%rd458]
|
||
s16078 L38574: @%p480 bra $L__BB1_534
|
||
```
|
||
|
||
Out-of-range lanes jump straight to `$L__BB1_534` (s16303) and skip (c)…(g).
|
||
|
||
**(c) row max over 5 shared values, s16085–s16100.**
|
||
|
||
```
|
||
s16085 L38582: ld.shared.u32 %r18915, [%r18964]
|
||
s16087 L38584: ld.shared.u32 %r18916, [%r18964+4]
|
||
s16091 L38588: ld.shared.u32 %r18936, [%r18964+16]
|
||
s16093 L38591: max.f16x2 %r18914,%r18915,%r18916
|
||
s16097 L38604: max.f16 %rs5266,%rs5267,%rs5268
|
||
s16099 L38611: mov.b32 %r18937, {%rs5269, %rs5269}
|
||
s16100 L38613: sub.f16x2 %r18923,%r18915,%r18937
|
||
```
|
||
|
||
Five `u32` (= 10 f16) are read per row and reduced in registers — **no warp
|
||
shuffle** — giving the row max, which is subtracted.
|
||
|
||
**(d) softmax, s16101–s16168.** Textbook f32 softmax over the row's 5 values:
|
||
|
||
```
|
||
s16103 L38621: mul.ftz.f32 %f1033, %f993, 0f3FB8AA3B // * log2(e)
|
||
s16104 L38622: ex2.approx.ftz.f32 %f995, %f1033
|
||
s16160 L38741: mov.f32 %f1032, 0f3F800000
|
||
s16161 L38742: div.approx.ftz.f32 %f1011, %f1032, %f1050
|
||
s16164 L38749: mul.f16x2 %r18938,%r18939,%r18952
|
||
```
|
||
|
||
`y = 2^((x−max)·log2e)`, `p = y · (1/Σy)` in f16, five f16x2 registers
|
||
(`%r18938 %r18941 %r18944 %r18947 %r18950`).
|
||
|
||
**(e) texture gather, s16169–s16266.** A 3×3 stencil of a colour texture:
|
||
|
||
```
|
||
s16184 L38789: cvt.rn.f32.s32 %f1051, %r18968
|
||
s16185 L38790: ld.param.v2.f32 {%f1052, %f1053}, [%rd459+136]
|
||
s16187 L38792: fma.rn.ftz.f32 %f1012, %f1052, %f1051, %f1056
|
||
s16191 L38796: ld.param.u64 %rd446, [%rd459+88]
|
||
s16193 L38799: tex.base.2d.v4.f16.f32 {%rs5294,%rs5295,%rs5296,%rs5297}, [%rd430, {%f1012,%f1013}]
|
||
s16195 L38803: fma.rn.f16 %rs5298,%rs5294,%rs5308,%rs5291
|
||
```
|
||
|
||
`%rd459 = param_0` (s16079), so the texture handle is **`param_0+88`**, the
|
||
scale/offset are **`param_0+136`** (two f32: `f1052·(k−0.5·?)`) and the
|
||
accumulator weights come from the softmax registers. Nine
|
||
`tex.base.2d.v4.f16.f32` are issued (s16193, 16205, 16217, 16229, 16234, 16240,
|
||
16252, 16258, 16263) for the 3×3 neighbourhood (clamped coordinates at
|
||
s16172–s16183 compute `(x−1, x, x+1)` and `(y−1, y, y+1)`), accumulated with
|
||
`fma.rn.f16` into 3 channels.
|
||
|
||
**(f) two integer-coordinate textures, s16267–s16270.**
|
||
|
||
```
|
||
s16267 L38971: ld.param.u64 %rd448, [%rd459+96]
|
||
s16268 L38973: tex.base.2d.v4.f16.s32 {%rs5438,%rs5439,%rs5440,%rs5441}, [%rd448, {%r549,%r550}]
|
||
s16269 L38975: ld.param.u64 %rd450, [%rd459+104]
|
||
s16270 L38977: tex.base.2d.v4.f16.s32 {%rs5442,%rs5443,%rs5444,%rs5445}, [%rd450, {%r549,%r550}]
|
||
```
|
||
|
||
**These are the `+96` / `+104` texture reads the brief asks about** — handles at
|
||
`param_0+96` and `param_0+104`, sampled at the *integer* pixel coordinate
|
||
`(%r549, %r550)`, returning two 4-channel f16 vectors.
|
||
|
||
**(g) combine, sigmoid gate, surface store, s16271–s16304.**
|
||
|
||
```
|
||
s16271 L38980: cvt.f32.f16 %f1030, %rs5446
|
||
s16272 L38983: mul.ftz.f32 %f1063, %f1030, 0fBFB8AA3B
|
||
s16273 L38984: ex2.approx.ftz.f32 %f1064, %f1063
|
||
s16274 L38985: add.ftz.f32 %f1065, %f1064, 0f3F800000
|
||
s16275 L38986: div.approx.ftz.f32 %f1031, %f1032, %f1065 // 1/(1+2^(-log2e·v))
|
||
s16278 L38996: sub.f16 %rs5449,%rs5448,%rs5447 // 1 - sigmoid
|
||
s16279 L39000: mul.f16 %rs5452,%rs5426,%rs5447 // acc * sigmoid
|
||
s16280 L39004: fma.rn.f16 %rs5495,%rs5438,%rs5449,%rs5452 // + tex96 * (1-sig)
|
||
s16285 L39024: set.nan.f16.f16 %rs5473,%rs5442,%rs5442 // + tex104 unless NaN
|
||
s16302 L39065: sust.b.2d.v4.b16.zero [%rd453, {%r18983,%r550}], {%rs5495,%rs5496,%rs5497,%rs5494}
|
||
```
|
||
|
||
`%rd453 = param_0+112` (s16090). The store address column is `%r549·8` (s16300
|
||
`shl.b32 %r18983, %r549, 3`), the row `%r550`; the fourth channel is the zero
|
||
literal `0x0000` (s16301 `mov.u16 %rs5494, 0`), so this is a **3-channel write
|
||
into a 4-channel surface**. The `set.nan`/`bra` triad (s16285–s16298) adds the
|
||
`+104` read only when it is not NaN.
|
||
|
||
**Inputs:** the 6 returned D registers, `param_0+{48,88,96,104,112,136}`, the
|
||
shared scratch, constants `0f3FB8AA3B`, `0fBFB8AA3B`, `0f3F800000`.
|
||
**Outputs:** none (the kernel returns, s16304).
|
||
|
||
---
|
||
|
||
## 4. Texture reads, surface writes, and the `+24` input
|
||
|
||
| what | where |
|
||
|---|---|
|
||
| `param_0+8` — input feature plane arena, read by phase 1's A loads | prologue s67…s324 (52 `ld.global.u32`) |
|
||
| `param_0+24` — the patch-gather source E1 reads | s2215 (`ld.param.u64 %rd149, [%rd1+-56]`, `%rd1 = param_0+80`) and the 24 `ld.global.v2.u16` at s2371…s5577 |
|
||
| `param_0+48` — the plane arena E25 scatters phase 25's output to | s15754 / s15755, 16 `st.global.u32` at s15780…s15959 |
|
||
| `param_0+88` — the colour texture handle | s16191, 9 `tex.base.2d.v4.f16.f32` at s16193…s16263 |
|
||
| `param_0+96` — texture handle #2 (integer coords) | s16267/s16268 |
|
||
| `param_0+104` — texture handle #3 (integer coords) | s16269/s16270 |
|
||
| `param_0+112` — the output surface | s16090, `sust.b.2d.v4.b16.zero` s16302 |
|
||
| `param_0+136` — two f32 texture coordinate scales | s16185 |
|
||
|
||
**The skip input at `+24` supplies the patch-gather source.** E1 reads it once
|
||
per lane per stencil position and never writes it; the buffer is addressed in
|
||
elements of 4 bytes (one f16x2 = 2 channels) with index
|
||
`clamp(row)·8·ex + clamp(col)·8 + clamp(t,0,7)` (s2245/s2250, L4674/L4679). It
|
||
is the *input* the patch expand tile is built from — the same operand the
|
||
window-1 patch is expanded out of.
|
||
|
||
**The surface writes** are `param_0+112` (E26, s16302), a 4-channel b16 surface
|
||
written with 3 live channels. The **plane-arena writes** are E25's 16
|
||
`st.global.u32` to `param_0+48` (s15780…s15959).
|
||
|
||
**Nothing reads `param_0+48` back** in this entry: `param_0+48` is the source of
|
||
exactly one `ld.param` (s15754) and never the base of a load. Likewise the
|
||
texture handles at `+88/+96/+104` are only ever `tex.*` operands.
|
||
|
||
## 5. The phase table, M/N/K checked
|
||
|
||
`rr_layer_spec.py dec1`'s phase table with the arithmetic `M/16 · N/8 · K/32 ==
|
||
mma` evaluated on every row (the brief's check):
|
||
|
||
| # | stmts | M | N | K | mma | M/16·N/8·K/32 | holds? | weight bytes (rel. `W`) | bias bytes |
|
||
|---|---|---|---|---|---|---|---|---|---|
|
||
| 1 | s403–s546 | 144 | 384 | 96 | 144 | 9·48·3 = 1296 | **no** | 0…11788 (+12288·`tid.z`) | 24576…24816 (+256·`tid.z`) |
|
||
| 2 | s7096–s7119 | 96 | 32 | 32 | 24 | 6·4·1 = 24 | yes | 25216…25740 (+18432·`tid.z`) | – |
|
||
| 3 | s7127–s7150 | 96 | 32 | 32 | 24 | 24 | yes | 27264…27788 (+18432·`tid.z`) | – |
|
||
| 4 | s7158–s7181 | 96 | 32 | 32 | 24 | 24 | yes | 26240…26764 (+18432·`tid.z`) | – |
|
||
| 5 | s7189–s7212 | 96 | 32 | 32 | 24 | 24 | yes | 28288…28812 (+18432·`tid.z`) | – |
|
||
| 6 | s7384–s7431 | 64 | 96 | 32 | 48 | 4·12·1 = 48 | yes | **none** | 29312…41100 (+18432·`tid.z`) |
|
||
| 7 | s11275–s11322 | 192 | 96 | 96 | 48 | 12·12·3 = 432 | **no** | **none** | – |
|
||
| 8 | s11506–s11521 | 32 | 64 | 32 | 16 | 2·8·1 = 16 | yes | 41600…43148 | – |
|
||
| 9 | s11538–s11553 | 32 | 64 | 32 | 16 | 16 | yes | 60032…61580 | – |
|
||
| 10 | s12397–s12412 | 64 | 64 | 64 | 16 | 4·8·2 = 64 | **no** | 62336…63884 | 64384…64432 |
|
||
| 11 | s12834–s12849 | 32 | 64 | 32 | 16 | 16 | yes | 64448…65996 | – |
|
||
| 12 | s12918–s12933 | 64 | 64 | 64 | 16 | 64 | **no** | 66624…68172 | 68672…68720 |
|
||
| 13 | s13242–s13257 | 32 | 64 | 32 | 16 | 16 | yes | 68736…70284 | – |
|
||
| 14 | s13326–s13341 | 64 | 64 | 64 | 16 | 64 | **no** | 70784…72332 | 72832…72880 |
|
||
| 15 | s13650–s13665 | 32 | 64 | 32 | 16 | 16 | yes | 72896…74444 | – |
|
||
| 16 | s13734–s13749 | 64 | 64 | 64 | 16 | 64 | **no** | 74944…76492 | 76992…77040 |
|
||
| 17 | s14058–s14073 | 32 | 64 | 32 | 16 | 16 | yes | 77056…78604 | – |
|
||
| 18 | s14142–s14157 | 64 | 64 | 64 | 16 | 64 | **no** | 79104…80652 | 81152…81200 |
|
||
| 19 | s14466–s14481 | 32 | 64 | 32 | 16 | 16 | yes | 81216…82764 | – |
|
||
| 20 | s14550–s14565 | 64 | 64 | 64 | 16 | 64 | **no** | 83264…84812 | 85312…85360 |
|
||
| 21 | s14874–s14889 | 32 | 64 | 32 | 16 | 16 | yes | 85376…86924 | – |
|
||
| 22 | s14958–s14973 | 64 | 64 | 64 | 16 | 64 | **no** | 87424…88972 | 89472…89520 |
|
||
| 23 | s15282–s15297 | 32 | 64 | 32 | 16 | 16 | yes | 89536…91084 | – |
|
||
| 24 | s15366–s15381 | 64 | 64 | 64 | 16 | 64 | **no** | 91584…93132 | 93632…93680 |
|
||
| 25 | s15690–s15705 | 32 | 64 | 32 | 16 | 16 | yes | 93696…95244 | – |
|
||
| 26 | s15981–s15988 | 64 | 32 | 64 | 8 | 4·4·2 = 32 | **no** | 95744…96268 | 96768…96784 |
|
||
|
||
**Every phase whose `M/16·N/8·K/32` does not equal `mma` is one the tool also
|
||
marks `grid = band`**, and every phase that holds is marked `full`. The
|
||
mismatching rows are *not* off by a constant factor (P1 is ×9, P7 ×9, P10/12/…
|
||
×4, P26 ×4), so `M`, `N` and `K` cannot all be taken at face value for those
|
||
rows: `nA` counts distinct A *register quads*, which can be either distinct
|
||
`M`-tiles or distinct `K`-slices of the same tile, and the tool cannot tell them
|
||
apart. See §7/U1.
|
||
|
||
**Which phases read no weight operand:** **P6 and P7**. Both take `B` from
|
||
registers that the preceding epilogue packed (E5 packs P2/P3's D for P6's A and
|
||
B; E6 packs the probabilities and the transposed P5 for P7's A and B). Their
|
||
`bo` is `None` for every mma.
|
||
|
||
**Weight image extent as the tool measures it:** the largest resolved offset is
|
||
**96784** (s15980, 4 bytes wide) ⇒ **96788 bytes** (634 distinct slots). This
|
||
excludes the two `tid.z`-strided groups in §1.1: phase 1's B and bias are read
|
||
at `W + 12288·tid.z` / `W + 256·tid.z`, and phases 2…5's B and phase 6's bias at
|
||
`W + 18432·tid.z + 25216`. With `tid.z ∈ {0,1}` (E26 rejects `tid.z > 1` at
|
||
s16069 `setp.gt.u32 %p474, %r19008, 1`) the true image must reach at least
|
||
`18432 + 41100 = 59532`, which is inside 96788, so the two `tid.z` slices do not
|
||
extend the image beyond what the tool already sees. **(INFERENCE** — the tool
|
||
drops the `tid.z` term, so the per-window base cannot be read off it directly.)
|
||
|
||
## 6. The plane arena (param_0 + 48) and the per-launch tables
|
||
|
||
* `param_0+8` is read 52 times in the prologue (s67…s324) as the **input feature
|
||
plane** that builds phase 1's 9 A fragments. The index arithmetic is
|
||
`(clamp_r·ey + clamp_c)·4` style (`shl.b32 … 3`, `mad.lo.s32`, `mul.wide.u32 …,
|
||
4`, `add.s64 …, %rd35`): the arena is a `{ex, ey}`-wide f16/byte plane read
|
||
with 4-byte elements.
|
||
* `param_0+24` is the **patch-gather source** (see §4); it is read, never written.
|
||
* `param_0+48` is the **output plane arena**: written by E25's 16
|
||
`st.global.u32` (s15780…s15959), never read. There is no second `st.*` group
|
||
in the entry (the entry has exactly 16 `st.global` and 0 other global stores
|
||
before L38179).
|
||
* The only other `ld.param` sites are the 4 in the prologue (§1.1) and the 5 in
|
||
E26 (`[%rd459+88/96/104/112/136]`, s16191/16267/16269/16090/16185) plus
|
||
`[%rd8+48]` (s15754) and the two `param_0+40` re-reads (s15961/s15962).
|
||
|
||
There is **no per-launch table generated into the plane arena by this kernel**:
|
||
the plane arena is only written (E25) and the tables that look like per-launch
|
||
tables (the `W + ((laneid<<2)&12) + off` blocks at 25088, 62080, 66496) all live
|
||
in the *weight image*, not the arena. See §7/U5.
|
||
|
||
## 7. Uncharacterised
|
||
|
||
**U1. The true `M/N/K` of the nine banded phases (P1, P7, P10, P12, …, P24,
|
||
P26).** For those rows `M/16·N/8·K/32 ≠ mma` (§5). The reason is structural
|
||
and checkable: the accumulator chains of a banded phase mix *distinct* B
|
||
operands. For P10 the four chains are
|
||
|
||
```
|
||
s12397 D=%r11522 A=%r11574 B=%r11473 C=bias (bo 62336)
|
||
s12399 D=%r11542 A=%r11594 B=%r11477 C=%r11522 (bo 62848)
|
||
s12401 D=%r11562 A=%r11574 B=%r11481 C=bias (bo 63360)
|
||
s12403 D=%r11582 A=%r11594 B=%r11485 C=%r11562 (bo 63872)
|
||
```
|
||
|
||
— the same D accumulates products with B fragments from four *different*
|
||
512-byte blocks and two *different* A quads. Under the settled weight formula
|
||
(`n = 16·tt + 8·(b>>3) + (L>>2)`, verified on enc0) those four blocks span
|
||
`n = 0…63`, so `N = 64` is supported; but then the two-mma chains must be
|
||
summing over `M` or `K`, and neither M nor K is separately readable from the
|
||
fragment structure. For P1 the same argument gives `N = 384` (48 B fragments =
|
||
24 blocks × 16 columns) while the arithmetic needs `M/16 · K/32 = 3`.
|
||
*Missing:* either a run with a weight-image dump, or the `_prep` permutation
|
||
that produces the image, or the high-level layer definition. Until then I will
|
||
not assign M/K to these rows.
|
||
|
||
**U2. The identity of the 12288-byte score bias (P6's C).** E5 loads it as 24
|
||
tiles × 512 bytes at `W + 18432·tid.z + 29312`. Whether it is a relative
|
||
position bias, a learned score bias, or a per-lane constant cannot be decided
|
||
from the PTX: it is an opaque blob in a permuted weight image. *Missing:* the
|
||
image's layout description (the `_prep` permutation) or the model's parameter
|
||
names.
|
||
|
||
**U3. The identity of the per-column tables.** `W+25088…25200` (E1 gain),
|
||
`W+62080…62192` (E7), `W+62208…62320` (E9 gain), `W+66496…66608` (E10) are all
|
||
read with the same `W + ((laneid<<2)&12) + off` pattern and are 8 words per lane.
|
||
Their *role* is computable (E1/E9: multiply the inverse RMS; E7/E10: added to a
|
||
residual before the next GEMM) but their *names* are not. The GEMM biases at
|
||
`+24576`, `+64384`, `+68672`, `+72832`, `+76992`, `+81152`, `+85312`, `+89472`,
|
||
`+93632`, `+96768` are the same case.
|
||
|
||
**U4. E1's patch-expand geometry.** The 48 `st.shared` of §3/E1 are guarded by
|
||
row/col limits of 9 and index `(row·40 + col·8 + t)·4` into four 1600-byte
|
||
sub-arrays, but they are emitted as ~400 basic blocks with 438 labels and I could
|
||
not reduce the lane→(row,col) map of the 10×10 patch to a closed form. What is
|
||
established: the four sub-arrays, the 40-word row pitch, the `t` and `t|4`
|
||
column split, the two `v2.u16` cells, and the total 6400 bytes.
|
||
*Missing:* the source-level im2col/patchify expression, or a shared-memory dump.
|
||
|
||
**U5. Whether `param_0+8` is an input plane or a scratch table.** The prologue
|
||
reads it 52 times with 4-byte elements and never writes it, and E1 does not read
|
||
it at all (E1 reads `param_0+24`). So `+8` and `+24` are two different input
|
||
buffers; which one holds the noisy colour plane and which holds the feature
|
||
planes cannot be decided from the PTX. *Missing:* the host-side launch
|
||
arguments.
|
||
|
||
**U6. The `tid.z` strides.** Two different per-`tid.z` strides (12288 for phase
|
||
1, 18432 for phases 2…5 + phase 6's bias, 256 for phase 1's bias, 192 for
|
||
phase 1's bias in the companion layout) appear, and no later phase carries one.
|
||
Whether this means the weight image is a concatenation of per-window sub-images
|
||
or that `tid.z` selects a batch element whose weights are interleaved cannot be
|
||
decided from the PTX alone. Consequence: the "96788 bytes" of §5 is a lower
|
||
bound for the image, not necessarily its size.
|
||
|
||
**U7. The logical meaning of E26's final expression.** The tail computes
|
||
`out = acc·σ(v) + tex96·(1−σ(v)) + tex104` on the three channels gathered from
|
||
the 3×3 colour stencil and the two integer-coordinate textures. The PTX gives
|
||
the arithmetic exactly (§3/E26(g)) but not what model term it corresponds to.
|
||
*Missing:* the high-level source or a reference implementation.
|
||
|
||
**U8. E12/E14/…/E24's `%f989…%f992`.** These six epilogues re-`cvt` the four
|
||
constants but never re-`mov.f32` them, so they read the registers E10 wrote
|
||
(s12414–s12425). That is only valid if the register allocator kept them live
|
||
across phases 11…23; the PTX shows no redefinition, so it is consistent, but
|
||
nothing in the file asserts the interference-freedom. *Missing:* an SSA check.
|
||
|
||
## 8. Quick acceptance index
|
||
|
||
| requirement | where |
|
||
|---|---|
|
||
| every epilogue named and ranged | §2, §3 (E1…E26) |
|
||
| phase-1 epilogue's patch expand + norm | §3/E1 |
|
||
| the norm (`E1`, `E9`) | §3/E1, §3/E9 — raw sum of squares + 2⁻¹³, no mean, no 1/N |
|
||
| the attention (score, softmax, P·V) | §3/E5, E6, E7; §3/E9 (region diagram) |
|
||
| the activations | §3/E10 (materialises `%f989…%f992`), E12, E14, E16, E18, E20, E22, E24 |
|
||
| the patch expand | §3/E1 (48 `st.shared`), §7/U4 |
|
||
| the final epilogue, textures at +96/+104, surface | §3/E26, §4 |
|
||
| the texture reads at +96/+104 | §3/E26(f), s16268 / s16270 |
|
||
| the skip input at +24 | §3/E1 (s2215), §4 |
|
||
| the plane-arena writes | §3/E25 (s15780…s15959), §4, §6 |
|
||
| the phase table with M/N/K checked | §5 |
|
||
| weight image extent | §5 (96788 bytes, 634 slots) |
|
||
| which phases read no weight | §5 (P6, P7) |
|
||
| uncharacterised | §7 |
|