NelsaHost
search Login

Can I run Qwen2-VL 72B on NVIDIA Jetson AGX Orin 32GB?

cancel
Fail/OOM
This GPU doesn't have enough VRAM
GPU VRAM
32.0GB
Required
144.0GB
Headroom
-112.0GB

VRAM Usage

0GB 100% used 32.0GB

info Technical Analysis

NVIDIA Jetson AGX Orin 32GB cannot run Qwen2-VL 72B in this configuration. The model requires 144.0GB but only 32.0GB is available, leaving you 112.0GB short.

lightbulb Recommendation

Consider using a more aggressive quantization (Q4_K_M, Q3_K_M) to reduce VRAM requirements, or upgrade to a GPU with more VRAM. Cloud GPU services like RunPod or Vast.ai offer affordable options.

tune Recommended Settings

Batch_Size
None
Context_Length
None
Inference_Framework
llama.cpp or vLLM

help Frequently Asked Questions

Can I run Qwen2-VL 72B on NVIDIA Jetson AGX Orin 32GB? expand_more
NVIDIA Jetson AGX Orin 32GB (32.0GB VRAM) cannot run Qwen2-VL 72B (72.00B) which requires 144.0GB. You are 112.0GB short. Consider using a more aggressive quantization (like Q4_K_M or Q3_K_M) or upgrading to a GPU with more VRAM.
How much VRAM does Qwen2-VL 72B need? expand_more
Qwen2-VL 72B requires approximately 144.0GB of VRAM.
What performance can I expect? expand_more
Estimated None tokens per second.

GPU

memory
NVIDIA Jetson AGX Orin 32GB
32.0GB VRAM

AI Model

smart_toy
Qwen2-VL 72B (72.00B)
72.00B params

More with Jetson AGX Orin 32GB

Whisper Small Perfect Whisper Medium Perfect Whisper Base Perfect SDXL 1.0 Perfect