Fine-tuning an open-weights model is cheap enough now that almost anyone can try it. Rent a GPU for a weekend, gather a few hundred examples, run a training script, and you get a model that sounds like your product. That first run often looks great in the logs and disappointing in the chat window.
The gap is rarely mysterious. After watching enough runs fail, the same handful of causes show up again and again: a template mismatch nobody noticed, a learning rate borrowed from the wrong recipe, an evaluation that was really just a vibe check. Here are the mistakes worth fixing before you burn another weekend of compute.
Mistake 1: Ignoring how your data is actually rendered
Your model never sees your JSONL file. It sees a token sequence produced by a chat template, and that template is a contract between training and inference. If the two differ, you have taught the model to respond to prompts it will never receive in production.
Use the tokenizer's built-in template, or the exact one published with the model card, and render a few examples by hand before training starts. Look for the things that quietly break runs: doubled BOS tokens, a system prompt present during training but missing at inference, or a role name like assistant versus model that survives the conversion. Printing five rendered samples takes two minutes and prevents a week of confusion.
Mask the prompt tokens
Many trainers compute loss over the entire sequence by default, including the user's turn. That trains the model to generate plausible user messages, which is not what you want and dilutes the signal you care about. Mask the prompt so only the completion contributes to the loss, unless you are deliberately doing continued pretraining on raw text.
Mistake 2: Letting evaluation data leak into training
Random splits look harmless on paper and are wrong more often than you would expect. If two paraphrases of the same question land on opposite sides of the split, your eval score measures memorization, not capability. Do the boring hygiene work first:
- Deduplicate. Near-identical rows act like sample weights. Twenty copies of one example turn it into 5% of your effective dataset.
- Split by entity or topic, not by row. Group everything about one customer, product, or document together, then assign whole groups to train or test.
- Freeze the eval set. Write it once, then do not touch it while iterating. Adjusting eval examples until scores look good is just overfitting with extra steps.
- Check the label balance. If 90% of your examples are one class, a model that always predicts that class will look accurate and be useless.
Mistake 3: Hyperparameters copied from the wrong recipe
Pretraining learning rates will wreck a fine-tune. Unless you are adapting a tiny model on a very large corpus, you want something in the range of 1e-5 to 2e-5 for full fine-tuning, and roughly 1e-4 to 2e-4 for LoRA adapters, with a warmup covering a few percent of total steps.
Epochs are the other common casualty. Instruction-style datasets usually want one to three passes; past that, models start reproducing exact training phrasing and losing the ability to generalize. Watch a handful of generations at each checkpoint rather than trusting the loss number. For effective batch size, accumulate gradients instead of shrinking the batch, since a tiny batch combined with a high learning rate produces noisy updates that erase themselves.
LoRA settings that matter
Rank 8 to 16 is often enough for tone, format, and output structure. Tasks that require genuinely new behavior tend to benefit from rank 32 or 64. Alpha is commonly set to one or two times the rank. Targeting attention projections only is the standard starting point; adding the MLP projections changes more of the model and is worth trying when attention-only adapters plateau.
Mistake 4: Judging the run by the loss curve
Training loss falls even when the outputs get worse. A model can memorize your dataset's exact wording while becoming brittle, repetitive, or incapable of handling a request phrased slightly differently. The only useful signal is generated text scored against a held-out set.
Build a small harness of 50 to 200 prompts that covers normal cases, awkward edge cases, the refusals you do want, and the refusals you do not. Score structured outputs with exact match or a regex. For open-ended answers, use a written rubric and spot-check by hand, because automated graders tend to reward longer, more confident answers regardless of correctness.
Always keep a baseline
Run the untouched base model through the same harness using the same prompt. Without that number you cannot tell whether fine-tuning helped or whether your prompt was doing the work all along. Save every adapter checkpoint and compare them against each other, not just against the last one.
Mistake 5: Expecting fine-tuning to teach facts
Fine-tuning reliably shapes behavior: format, tone, task patterns, how a model structures an answer. It is an unreliable way to install factual knowledge, and a poor fit for anything that changes often. If your goal is "answer questions from this documentation," retrieval is the right tool. If your goal is "always answer in this schema, in this voice," fine-tuning is. Mixing the two produces a model that confidently invents details on anything outside its training data.
Mistake 6: Leaving inference to the end
A model trained in fp16 and served quantized to 4-bit can behave noticeably differently. So can a model that saw a system prompt during training and gets none at inference. Test in the configuration you plan to ship: the same quantization, the same serving stack, the same sampling parameters, the same stop tokens. Check that your maximum output length is not silently truncating answers, and that your system prompt travels with the adapter.
A pre-flight checklist
Run through this before your next training job, and keep it next to the config file:
- Render several examples through the exact inference template and read them.
- Deduplicate, split by topic or entity, and freeze the eval set.
- Score the base model on the eval harness.
- Start with LoRA rank 16, learning rate 1e-4, one to three epochs, warmup at a few percent.
- Confirm prompt tokens are masked out of the loss.
- Evaluate generations every few hundred steps, and keep every checkpoint.
- Pick the best checkpoint by eval, then read twenty outputs yourself.
- Re-test at the quantization and serving settings you will actually deploy.
Expect two to four rounds of iteration before a run feels finished. When the eval score refuses to move, the cause is almost always the data or the template rather than the learning rate — so go back and read your rendered examples again. That single habit has saved more fine-tuning projects than any hyperparameter sweep.
Photo: panumas nikhomkhai / Pexels
