NelsaHost

Hardware AI Models Compatibility Compare

search Login

Can I run Mixtral 8x22B (Q4_K_M (GGUF 4-bit)) on NVIDIA RTX 4090?

cancel

Fail/OOM

This GPU doesn't have enough VRAM

GPU VRAM

24.0GB

Required

70.5GB

Headroom

-46.5GB

VRAM Usage

0GB 100% used 24.0GB

info Technical Analysis

NVIDIA RTX 4090 cannot run Mixtral 8x22B (141.00B) in this configuration. The model requires 70.5GB but only 24.0GB is available, leaving you 46.5GB short.

lightbulb Recommendation

Consider using a more aggressive quantization (Q4_K_M, Q3_K_M) to reduce VRAM requirements, or upgrade to a GPU with more VRAM. Cloud GPU services like RunPod or Vast.ai offer affordable options.

tune Recommended Settings

Batch_Size

None

Context_Length

None

Inference_Framework

llama.cpp or vLLM

help Frequently Asked Questions

Can I run Mixtral 8x22B (141.00B) on NVIDIA RTX 4090? expand_more

NVIDIA RTX 4090 (24.0GB VRAM) cannot run Mixtral 8x22B (141.00B) which requires 70.5GB. You are 46.5GB short. Consider using a more aggressive quantization (like Q4_K_M or Q3_K_M) or upgrading to a GPU with more VRAM.

How much VRAM does Mixtral 8x22B (141.00B) need? expand_more

Mixtral 8x22B (141.00B) requires approximately 70.5GB of VRAM.

What performance can I expect? expand_more

Estimated None tokens per second.