The AMD RX 7900 XTX, with its 24GB of GDDR6 VRAM and 0.96 TB/s memory bandwidth, is well-suited for running the Gemma 2 2B model, especially when quantized to INT8. The INT8 quantization significantly reduces the model's memory footprint to approximately 2.0GB, leaving a substantial 22GB VRAM headroom. This large headroom allows for higher batch sizes and longer context lengths without exceeding the GPU's memory capacity. The RDNA 3 architecture, while lacking dedicated Tensor Cores like NVIDIA GPUs, can still leverage its compute units for efficient matrix multiplications, which are crucial for LLM inference. The estimated tokens/sec of 63 suggests reasonable performance, but the actual throughput will depend on factors such as the specific inference framework used and other system bottlenecks.
Given the ample VRAM headroom, experiment with increasing the batch size to potentially improve throughput. Start with a batch size of 32, as suggested, and gradually increase it until you observe diminishing returns or encounter memory-related errors. Consider using an inference framework optimized for AMD GPUs, such as llama.cpp with the appropriate ROCm backend or ONNX Runtime, to maximize performance. Profile your application to identify any CPU bottlenecks that might be limiting the GPU's utilization. For the context length, 8192 tokens should be easily handled, but monitor memory usage if increasing it further.