Performance Tuning
Performance tuning guide for the Mini-Inference Engine.
Key Metrics
- GFLOPS: FLOPs / (time_ms * 1e6)
- Memory Bandwidth Utilization: Actual / Peak bandwidth
- Arithmetic Intensity: FLOPs / Bytes
Block Size Selection
| Parameter | Description | Typical |
|---|---|---|
| BM | Block M dimension | 64-256 |
| BN | Block N dimension | 64-256 |
| BK | Iteration K dimension | 8-32 |
| TM | Thread M dimension | 4-8 |
| TN | Thread N dimension | 4-8 |
Architecture-Specific Tips
- Volta (SM 7.0): BM=128, BN=128
- Turing (SM 7.5): Consider FP16
- Ampere (SM 8.0): Use
cp.async - Ada (SM 8.9): Maximize compute density
Profiling Tools
bash
cd build/default/bin
nsys profile -o report ./mini_inference_benchmark
ncu --set full -o report ./mini_inference_benchmark