6.S894

6.S894 Accelerated Computing provided a solid introduction to programming on GPUs with CUDA. The first three labs taught the fundamentals, starting out with implementing Mandelbrot on a CPU with AVX512 vectorization and on a GPU to learn the basics of thinking about parallel computation and to introduce the class’s GPU: NVIDIA RTX 4000 Ada. The next three built up matmul through tiling, overlapping memory and compute, and using the tensor cores. The second half of the labs explored more breadth like a parallel hierarchical scan and implementing reduce-scatter and all-gather on a 2x2 mesh of Google TPUv5ps in Pallas.

The coolest thing, though, was redoing matmul, now on an H100. We got to use more advanced features like TMA (Tensor Memory Accelerator) and wgmma (warp group-level matrix multiply) and ended up getting 95% of cuBLAS. Check out the full write up: H100 Matmul.

Taking this class after 6.1060 (it’s a prereq) helped solidify a lot more of the techniques of performance engineering. GPUs give you a lot more direct control over how your code is actually run and lets you make explicit choices over how to interact with the compute and memory units and see visible results.

Overall, this was a great class which definitely shaped my interest in GPU programming. Here is the class website. I would also recommend checking out its Resources tab which has info on SASS, PTX, TMA, and of course links GPU Mode.