Raising a small LLM on a noisy desk

I’d been putting it off for weeks. Fine-tuning a 7B model sounded like the kind of thing that needed a real GPU, a real plan, and some sense of professionalism — three things in short supply on a Tuesday night. So I started anyway. Here are the notes.

The corpus is small — about 6 years of personal correspondence, exported as plaintext, tagged loosely. Roughly 1.4M tokens after cleaning. Small enough that I can re-derive it on my laptop in about ninety seconds, large enough that a model can find a voice in it.

”Teaching a small model your own writing voice is mostly an exercise in deciding which parts of yourself you actually want to keep.”

§1 · The setup

One 4090, axolotl, qLoRA, 4-bit, batch size 1, gradient accumulation 16. I ran the first epoch with my partner asleep next to me; the fans woke her up exactly twice.

$ axolotl train ./homecanon.yaml --deepspeed off
[axolotl] loaded base: meta-llama/Llama-3-8B
[axolotl] training set: 1,432,118 tokens
[axolotl] estimated wall: 03:42
[axolotl] checkpoint every 200 steps
> step 200 | loss 1.84 | lr 2e-5 | gpu 23.1G/24G

§2 · What broke

Three things, in order. The first was my fault, the second was a known issue, the third I’m still not sure about. I’ll write each up below — but the short version is: trust the loss curve more than the eval samples, and never let a friend pick the prompts.

More to come. I’ll keep the weights private but the recipe is on forge.tinkerlab.dev/homecanon.