Technical notes on learning systems, GPU kernels, embodied intelligence, and research engineering.https://shivampr21.com/en-gbhttps://shivampr21.com/posts/flash-bwd-pll-14-4-2025-kernelized/https://shivampr21.com/posts/flash-bwd-pll-14-4-2025-kernelized/Mapping the backward pass into dependency graphs, loop-carried constraints, and parallel execution.Mon, 14 Apr 2025 00:00:00 GMTFlashAttentionTransformersAttentionComputeAutogradParallelismCUDAhttps://shivampr21.com/posts/flash-fwd-pll-14-4-2025-kernelized/https://shivampr21.com/posts/flash-fwd-pll-14-4-2025-kernelized/A dependency-led path from the forward-pass equations toward the CUDA programming model.Mon, 14 Apr 2025 00:00:00 GMTFlashAttentionTransformersAttentionComputeAutogradParallelismCUDAhttps://shivampr21.com/posts/flash-30-3-2025-kernelized/https://shivampr21.com/posts/flash-30-3-2025-kernelized/A first-principles derivation of the online, numerically stable FlashAttention forward pass.Sun, 30 Mar 2025 00:00:00 GMTFlashAttentionTransformersAttentionComputeAutogradMATHhttps://shivampr21.com/posts/flash-bwd-30-3-2025-kernelized/https://shivampr21.com/posts/flash-bwd-30-3-2025-kernelized/Deriving attention gradients from tensor differentiation to a computable backward pass.Sun, 30 Mar 2025 00:00:00 GMTFlashAttentionTransformersAttentionComputeAutogradMATHhttps://shivampr21.com/posts/max-30-3-2025-kernelized/https://shivampr21.com/posts/max-30-3-2025-kernelized/The forward and backward mathematics of a max-reduction kernel using indexed tensor notation.Sun, 30 Mar 2025 00:00:00 GMTAIComputeAutogradMATHhttps://shivampr21.com/posts/softmax-30-3-2025-kernelized/https://shivampr21.com/posts/softmax-30-3-2025-kernelized/A compact derivation of softmax forward and backward operations for later kernel work.Sun, 30 Mar 2025 00:00:00 GMTAIComputeAutogradMATH