Learning Resources
A curated list of resources for learning CUDA programming and GPU kernel optimization.
Suggested reading path
If you are using TensorCraft-HPC as a learning project or interview artifact, this order works well:
- Read the project whitepaper to understand the repository's design intent.
- Use this page to branch outward into the core NVIDIA references and neighboring open-source projects.
- Return to the repository's architecture overview and kernel atlas with that external context in mind.
This makes the project easier to compare honestly against production libraries and research-grade implementations.
Official NVIDIA Resources
Documentation
- CUDA C++ Programming Guide — The authoritative reference for CUDA programming
- CUDA Best Practices Guide — Optimization strategies and common pitfalls
- CUDA Profiler Tools Interface (CUPTI) — For profiling CUDA applications
Libraries
- cuBLAS — Dense linear algebra
- cuDNN — Deep learning primitives
- cuSPARSE — Sparse linear algebra
- NCCL — Multi-GPU communication
Tools
- Nsight Compute — Kernel profiling and analysis
- Nsight Systems — System-wide profiling
- NVIDIA Visual Profiler — Legacy GUI profiler
Open Source Projects
Kernel Libraries
| Project | Focus | Difficulty |
|---|---|---|
| CUTLASS | GEMM, Tensor Cores | Advanced |
| FlashAttention | Attention | Advanced |
| xFormers | Attention, Memory | Intermediate |
| Triton | DSL for kernels | Intermediate |
| DeepSpeed | Training optimization | Advanced |
How these projects relate to TensorCraft-HPC
| Project | Why compare it | What TensorCraft-HPC emphasizes instead |
|---|---|---|
| CUTLASS | Canonical CUDA GEMM / Tensor Core engineering | Simpler learning path and clearer optimization narration |
| FlashAttention | Reference-quality attention implementation | Easier-to-follow explanation of tiling and memory trade-offs |
| Triton | Alternative kernel authoring model | Direct C++/CUDA control and closer-to-metal educational examples |
| xFormers / DeepSpeed | Real-world training-system context | Focused operator learning rather than full-stack training infrastructure |
Educational
| Project | Description |
|---|---|
| CUDA Mode | CUDA learning resources |
| GPU Mode | GPU programming tutorials |
| Awesome CUDA | Curated CUDA resources |
Books
GPU Programming
- Programming Massively Parallel Processors — David B. Kirk, Wen-mei W. Hwu
- The classic textbook for GPU computing
- CUDA by Example — Jason Sanders, Edward Kandrot
- Practical introduction to CUDA
- Professional CUDA C Programming — John Cheng, Max Grossman, Phil McGachey
- Advanced CUDA techniques
Computer Architecture
- Computer Architecture: A Quantitative Approach — Hennessy & Patterson
- Understanding memory hierarchies and parallelism
Online Courses
- NVIDIA Deep Learning Institute — Official NVIDIA courses
- CMU 15-418: Parallel Computer Architecture — Excellent course on parallelism
- MIT 6.172: Performance Engineering — Software performance optimization
Key Concepts
Memory Hierarchy
Execution Model
Optimization Priority
- Maximize Parallelism — Enough threads to hide latency
- Coalesced Memory Access — Adjacent threads access adjacent memory
- Shared Memory Usage — Reduce global memory traffic
- Bank Conflict Avoidance — Ensure shared memory efficiency
- Occupancy Tuning — Balance registers, shared memory, threads
What to borrow into this project
When expanding TensorCraft-HPC, the most valuable ideas to absorb from the surrounding ecosystem are:
- from CUTLASS: disciplined tiling vocabulary and Tensor Core decomposition patterns
- from FlashAttention: memory-aware storytelling and IO-driven reasoning
- from Triton: clear operator-level benchmarking habits and compact examples
- from Nsight tooling: evidence-first performance explanations instead of intuition-led guesses
What to borrow, what to resist
The repository becomes stronger when it borrows methods, not when it imitates style blindly.
Borrow
- disciplined vocabulary for tiling, memory traffic, and hardware capability
- benchmark methodology that exposes caveats and baselines
- compact operator examples that map cleanly to the public API
Resist
- production-only abstractions that hide the optimization path
- feature sprawl that makes the educational story harder to follow
- benchmark claims separated from tooling, workload shape, or reference library
This tension is intentional. TensorCraft-HPC should learn from stronger systems without becoming another opaque production stack.
Performance Metrics
| Metric | Description | Target |
|---|---|---|
| Throughput | Operations per second | Roofline limit |
| Latency | Time per operation | Minimal |
| Occupancy | Active warps / Max warps | 50-100% |
| Memory Bandwidth | Bytes transferred / second | ~90% peak |
| Compute Efficiency | Achieved / Peak FLOPS | >80% for GEMM |
Common Pitfalls
Memory Coalescing
Non-coalesced memory access can reduce bandwidth by 10-32x. Always ensure adjacent threads access adjacent memory addresses.
Shared Memory Bank Conflicts
When multiple threads in a warp access the same bank, access is serialized. Use padding or access patterns to avoid.
Branch Divergence
Divergent branches within a warp execute both paths sequentially. Minimize control flow divergence.
Profiling First
Always profile before optimizing. Use Nsight Compute to identify actual bottlenecks rather than guessing.