What is LoRA and why did it change fine-tuning economics?
Any role touching model training or deployment asks about PEFT — LoRA is the canonical answer.
Full fine-tuning updates all billions of parameters. GPU memory and checkpoint size scale linearly.
LoRA freezes base weights W and learns a low-rank update: W + BA where A ∈ R^(r×d), B ∈ R^(d×r), r ≪ d. Only A, B are trained.
Why it works: the "intrinsic rank" of fine-tuning updates is low — they live in a small subspace. r=8 or r=16 is usually enough.
Benefits: checkpoint is MBs not GBs, training fits on one GPU where full FT needs many, can hot-swap adapters per task.
Extensions: QLoRA quantizes base model to 4-bit, fine-tunes LoRA on top — 70B fine-tune on a single consumer GPU.
Full fine-tuning of a 70B model needs hundreds of GB of GPU memory for optimizer states and gradients. LoRA's insight: the update you actually need from fine-tuning lives in a low-rank subspace, so instead of updating the full W, learn a low-rank W + BA where rank r is maybe 8 or 16. Freeze base, train only A and B. Checkpoint goes from 140GB to 40MB. Because the update is additive and small, you can keep multiple adapters for different tasks and hot-swap them at inference. QLoRA takes it further — quantize the base to 4-bit and train LoRA on top, which makes 70B fine-tuning possible on a single 24GB GPU. For most task-specific adaptation (tone, domain, instruction format) LoRA matches full FT within 1-2% of eval metrics while costing 100x less.
Target the attention projection matrices (q_proj, v_proj) first — biggest bang for buck.
Rank too low: underfit. Rank too high: overfit + defeats the purpose. r=8 or 16 is a sane default.
HireMe AI (即答侠) lays out the full-FT vs LoRA vs QLoRA memory/cost table interviewers love to see.
Yes — compute W' = W + BA once and deploy W'. Zero runtime overhead after merge.
Fundamental capability changes (new languages, new modalities) usually need full fine-tuning or pre-training. LoRA is best for style/task/domain adaptation.
HireMe AI listens to your interview in real-time, detects questions, and generates instant answer suggestions — so you never freeze up again.
Try HireMe AI Free