Skip to main content

GPU sharing on orca

orca has two NVIDIA H100 NVL GPUs (~95 GB VRAM each, 190 GB total), shared between the lab's LLM inference service and everyone's research jobs (training runs, GPU-accelerated experiments). There is no job scheduler — sharing works by the conventions on this page.

What's already running

The LLM service (LiteLLM → llama-swap) uses the GPUs elastically:

  • Every LLM model unloads after 1 idle hour. If nobody has used the service for an hour, GPU usage drops to near zero and essentially all 190 GB is available for your jobs. The next LLM request just pays a ~1-minute reload.
  • While in use: the main chat model takes ~71 GB (split across both cards), and loading one of the largest models can temporarily bring total usage to ~170 GB. A large-model load simply fails if your job holds the space — during a reserved run that's expected, not an emergency.
  • Speech-to-text (WhisperX) holds ~4 GB while its container is up.

Practical meaning: nvidia-smi will show usage that varies over time and isn't anyone's job. Free VRAM is genuinely free to use.

Rules of thumb (always apply)

  1. Check before you allocate. Run nvidia-smi and look at both memory and utilization on each card. Don't start a large job on a card that's busy.
  2. Pin your job to one card with CUDA_VISIBLE_DEVICES=0 (or =1) unless you actually need both. This keeps the other card clean for the LLM service and other users. (PyTorch needs no other configuration — it allocates memory incrementally as it goes.)
  3. Checkpoint long runs. There is no preemption, but the machine can be rebooted for maintenance; anything running longer than ~12 h should be resumable.
  4. CPU-only work needs none of this. scikit-learn, pandas, plain NumPy don't touch the GPU — orca has 393 GB of RAM; go wild.

Reservations

Small jobs (up to ~40 GB VRAM, a few hours): just run them — no reservation needed. That fits comfortably alongside the LLM service even at its busiest.

Anything bigger or longer: post in the Zulip #compute-gpu channel, stating which GPU(s), roughly how much VRAM, and until when. That's the whole process — first post wins, coordinate in-thread if two runs collide.

During a reservation, the reservation wins. The largest LLM models may fail to load for other users while a big reserved run holds the space; the main chat model keeps working either way (it reloads into whatever room remains after an idle unload). If you need the big LLM models and a reservation is active, wait it out or ask in the thread.

Back to top