How to Fine-Tune a Small LLM on a Single GPU

Most teams assume fine-tuning needs a multi-GPU cluster. For a small-to-mid model (1B–14B parameters), it usually doesn't — a single 24GB consumer GPU is enough if you use a parameter-efficient method like LoRA. This is the walkthrough I wish existed when I fine-tuned my first model on a single RTX 4090.

Why fine-tune instead of prompting or RAG?

Before touching training code, it's worth being honest about when fine-tuning actually earns its cost. It's the right call when:

• You need a consistent output format the base model won't reliably follow with prompting alone
• Your domain vocabulary or task structure is narrow and repetitive (support ticket classification, structured extraction)
• Latency or cost rules out a large hosted model, and a small fine-tuned model can match its accuracy on your specific task

If your problem is "the model doesn't know about our internal docs," that's a RAG problem, not a fine-tuning problem — fine-tuning teaches a model to behave a certain way, not to memorize facts reliably.

1. Prepare the dataset

Data quality matters more than model size or training steps. For a classification-style fine-tune, aim for:

{"instruction": "Classify the support ticket priority.", "input": "Checkout page returns 500 for all EU customers.", "output": "P0 - Critical"}
{"instruction": "Classify the support ticket priority.", "input": "Button color looks slightly off in dark mode.", "output": "P3 - Low"}

A few hundred well-labeled examples per class beats a few thousand noisy ones. Hold out 10–15% as an evaluation set you will not train on — you'll need it later to prove the fine-tune actually helped.

2. Pick a base model and method

For a single-GPU setup, QLoRA (quantized LoRA) is the practical default — it 4-bit quantizes the base model and trains small low-rank adapter weights on top, cutting memory use dramatically with a small accuracy trade-off.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)
 
model = AutoModelForCausalLM.from_pretrained(
    "base-model-7b",
    quantization_config=bnb_config,
    device_map="auto",
)
 
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
 
model = get_peft_model(model, lora_config)

3. Train

Keep the run short and watch the eval loss, not just training loss — a small dataset overfits fast.

from transformers import TrainingArguments, Trainer
 
args = TrainingArguments(
    output_dir="./out",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=3,
    fp16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="epoch",
)
 
trainer = Trainer(model=model, args=args, train_dataset=train_ds, eval_dataset=eval_ds)
trainer.train()

4. Evaluate honestly

Run the held-out set through both the base model and the fine-tuned model with identical prompts, and score them the same way — exact match, F1, or an LLM-judge rubric, depending on the task. If the fine-tune doesn't beat the base model by a meaningful margin on this held-out set, it's not done yet — or it wasn't the right lever for this problem.

Where this breaks down

QLoRA on one GPU works well up to roughly 13B-parameter base models with a 24GB card. Past that, you're either quantizing more aggressively (with more accuracy loss) or you need multiple GPUs. For most product features, a well-tuned small model beats a barely-fine-tuned large one anyway — smaller, cheaper to serve, and easier to iterate on.

This is a structural walkthrough — swap in your own dataset, base model, and evaluation rubric before running it against a real task.