NelsaHost
search Login

Can I run Llama 3.1 8B (Q4_K_M (GGUF 4-bit)) on NVIDIA A100 80GB?

check_circle
Perfect
Yes, you can run this model!
GPU VRAM
80.0GB
Required
4.0GB
Headroom
+76.0GB

VRAM Usage

0GB 5% used 80.0GB

Performance Estimate

Tokens/sec ~93.0
Batch size 32
Context 128000K

info Technical Analysis

GPU

memory
NVIDIA A100 80GB
80.0GB VRAM

AI Model

smart_toy
Llama 3.1 8B (8.00B)
8.00B params

Alternative Quantizations

q3_k_m Perfect INT8 (8-bit Integer) Perfect Perfect

More with A100 80GB

Qwen 2.5 7B Perfect Qwen 2.5 7B Perfect Qwen 2.5 7B Perfect Qwen 2.5 7B Perfect