FlashAttention

FlashAttention uses tiling and online softmax to compute attention without materializing the full N×N attention matrix in HBM.

Standard Attention Problem

FlashAttention Data Flow

Total HBM access: O(N) vs O(N²) for standard attention.

References