| forward peak memory/batch size | |
| generate peak memory/batch size | |
| generate throughput/batch size | |
| forward latency/batch size | |
| The benchmarks indicate AWQ quantization is the fastest for inference, text generation, and has the lowest peak memory for text generation. |