Serve a model

Start a GPU, run a GGUF model with llama.cpp, and use it through an OpenAI-compatible API from curl, Python, or any OpenAI-compatible tool.

hi compute serve rents a GPU, starts llama.cpp’s server with a GGUF model, waits until it answers, and gives you an OpenAI-compatible URL. It takes a tested recipe or any GGUF repository on the Hugging Face Hub.

With a recipe

Recipes pick the model file, hardware, context length, and server flags that have been tested together:

hi compute serve qwen3.8-flash-next --on colab --max 1h

Try a 125B model on a 96 GB GPU walks through this one.

With any GGUF model

Name a Hugging Face repository that contains GGUF files, a quantization that exists in it, and hardware with enough memory:

hi compute serve unsloth/Qwen3-8B-GGUF --quant Q4_K_M --gpu L4 --max 1h
hi compute serve unsloth/Qwen3-0.6B-GGUF --quant Q4_K_M --gpu t4-small --max 30m

Choose hardware whose GPU memory is at least the model file’s size plus a few GB for the context. A 5 GB Q4_K_M file fits a 16 GB T4 comfortably; a 90 GB file needs a 96 GB GPU.

What happens on each provider

Colab

1. hi starts the machine, uploads a setup script over SSH, and runs it in the background, so a dropped connection never stops setup.
2. The script builds llama.cpp with CUDA and downloads the model in parallel. hi prints each step as it changes:

     [19:57:04] building llama.cpp and downloading unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q3_K_XL  (model on disk: 4.5G)
     [19:58:58] llama.cpp built, downloading …  (model on disk: 39G)
     [20:01:24] loading model  (model on disk: 84G)
     [20:03:57] ready
   OpenAI-compatible API: http://127.0.0.1:8080/v1   model: qwen3.8-flash-next
   

3. It then tunnels the API to http://127.0.0.1:8080/v1 and stays in the foreground. Press Ctrl+C to close the tunnel; hi asks whether to stop the machine.

If setup fails or you close the terminal, run the same serve command with the same --name again: the script skips what is already done.

Hugging Face

1. hi starts one job that is the server, using llama.cpp’s official CUDA image. llama-server downloads the model itself.
2. When the server answers, hi prints an HTTPS URL and returns:

     [20:34:55] downloading and loading unsloth/Qwen3-0.6B-GGUF:Q4_K_M
     [20:35:36] ready
   OpenAI-compatible API: https://6ab97de2…--8000.hf.jobs/v1   model: qwen3-0.6b-gguf
   It needs your hf token as the API key, and stops after 30m.
   

3. The URL works from anywhere, but only with a Hugging Face token that can read the account that pays; without one, it answers 401. Use your token as the API key.

To use it only from your laptop instead, tunnel it: hi compute tunnel <name> 8000:8080.

Use the API

The server speaks the OpenAI chat completions API.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3.8-flash-next", "messages": [{"role": "user", "content": "Hello"}]}'

For a Hugging Face URL, add the token as a header:

export HF_TOKEN=$(hf auth token)
curl https://<job>--8000.hf.jobs/v1/models -H "Authorization: Bearer $HF_TOKEN"

From Python with the openai package:

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",  # or the https://…hf.jobs/v1 URL
    api_key=os.environ.get("HF_TOKEN", "unused"),
)
reply = client.chat.completions.create(
    model="qwen3.8-flash-next",
    messages=[{"role": "user", "content": "Explain compute units in one sentence."}],
)
print(reply.choices[0].message.content)

Most OpenAI-compatible tools accept the same two settings, often as environment variables:

export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=unused      # or your Hugging Face token for an hf.jobs URL

Reasoning models such as Qwen3 think before answering. The thinking arrives in reasoning_content and counts toward max_tokens, so give them room (for example "max_tokens": 2000) or the answer may be empty.

Watch and troubleshoot

hi compute logs <name>            # setup, download, and server logs
hi compute logs <name> --follow
hi compute status <name>

Stop

A model server keeps costing money until it stops:

hi compute stop <name>