Shivam PandeyAI Research EngineerIM

02 / Writing

Working notes from
inside the machine.

Derivations, implementations, and experiments written to expose the path—not only the result. New long-form work will be published through Immortal Machine; the KERNELIZED archive remains here and fully readable.

Series 001

KERNELIZED

Six notes on reduction, softmax, and FlashAttention: from tensor notation and autodiff to dependency graphs and parallel execution.

  1. 06

    FlashAttention · Transformers · Attention

    FlashAttention Kernel: Backward Pass (Parallelism)

    Mapping the backward pass into dependency graphs, loop-carried constraints, and parallel execution.
  2. 05

    FlashAttention · Transformers · Attention

    FlashAttention Kernel: Forward Pass (Parallelism)

    A dependency-led path from the forward-pass equations toward the CUDA programming model.
  3. 04

    FlashAttention · Transformers · Attention

    FlashAttention Kernel: Backward Pass (MATH)

    Deriving attention gradients from tensor differentiation to a computable backward pass.
  4. 03

    FlashAttention · Transformers · Attention

    FlashAttention Kernel: Forward Pass (MATH)

    A first-principles derivation of the online, numerically stable FlashAttention forward pass.
  5. 02

    AI · Compute · Autograd

    SoftMax Kernel: Forward and Backward Pass (MATH)

    A compact derivation of softmax forward and backward operations for later kernel work.
  6. 01

    AI · Compute · Autograd

    Max Kernel: Forward and Backward Pass (MATH)

    The forward and backward mathematics of a max-reduction kernel using indexed tensor notation.