Tuning Llama for a Small Model

Getting a model running is covered in Setting Up Inference. This page is about what you pass to Llama(...) afterwards — which settings decide whether it runs at all, which only change the output, and how to tell them apart.

The distinction matters because they fail differently. A memory setting that is too ambitious fails loudly, at load, before a single token is generated. A sampling setting that is wrong just gives you worse answers.

The memory budget¶

Three things occupy VRAM:

1. The model weights — fixed by which file you chose.

2. The KV cache — scratch space for the conversation so far. Set by n_ctx, allocated up front, and the usual surprise.

3. Working memory for processing the prompt, influenced by n_batch.

On a shared T4 there is 16GB and two people drawing on it. Plan around 7GB, not 16.

n_ctx — the one that catches people¶

n_ctx is how many tokens the model can hold in view. It looks free. It is not: the KV cache grows linearly with it, and the whole allocation happens when the model loads.

n_ctx = 512   # the default: small
n_ctx = 0     # means "use the model's full trained length"

Set it to what your prompts actually need, with headroom. Question-answering over a paragraph might need 2,048. Summarising a long PDF might need 8,192. Reach for 32,768 only after measuring that you need it.

When the model loads, llama.cpp prints what it allocated — look for the lines reporting KV cache and buffer sizes. That is the authoritative number, and it moves when you change n_ctx. Halve n_ctx, reload, and watch it halve.

n_gpu_layers — everything on the GPU, or split¶

n_gpu_layers = 0    # the default: CPU only
n_gpu_layers = -1   # every layer on the GPU
n_gpu_layers = 20   # the first 20 layers on the GPU, the rest on CPU

Put everything on the GPU — -1 — when the weights plus the KV cache fit in your share of the card. For a quantised 1–2B model at a modest n_ctx they will, and it is several times faster. This is the case you want.

Split the work when they do not fit. Offloading only some layers runs the remainder on CPU, and every token crosses that boundary, so it is substantially slower — but it is the difference between slow and not running at all. Start high and come down until it loads: try -1, then 24, then 16.

Splitting is also a considerate choice when someone else is working on the other half of the card and you would otherwise crowd them out.

n_batch¶

How many prompt tokens are processed at once (default 512). It affects memory during the prompt phase, not generation. If a model loads happily and then runs out of memory on a long prompt, lower this before touching anything else.

The settings that only change the output¶

None of these affect whether the model fits.

A starting point¶

from llama_cpp import Llama

llm = Llama(
    model_path="/home/jovyan/shared/models/gguf/your-model.gguf",
    n_gpu_layers=-1,   # all layers on the GPU
    n_ctx=4096,        # raise only if your prompts need it
    n_batch=512,
    seed=42,
)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain gradient descent in two sentences."}],
    temperature=0.7,
    max_tokens=256,
)

Change one thing at a time. If it stops loading, the last thing you changed was almost certainly n_ctx or n_gpu_layers.

When it fails¶