LoRA and QLoRA Explained: Fine-Tuning Big Models on Small Budgets

This block calculates the total VRAM needed to full-fine-tune a 7B model, breaking down the three components that consume GPU memory.

• params = 7e9 — sets the parameter count to 7 billion (7 × 10⁹).
• bytes_per_param = 4 — specifies that float32 (32-bit floating point) uses 4 bytes per parameter, the standard precision for training.
• model_weights = params * bytes_per_param / 1e9 — computes the model weights in GB: 7e9 × 4 / 1e9 = 28 GB. This is the cost of just loading the model.
• gradients = params * bytes_per_param / 1e9 — computes the gradients in GB (another 28 GB). During backpropagation, every parameter needs a gradient of the same size, so this doubles the memory.
• optimizer_states = params * bytes_per_param * 2 / 1e9 — computes the optimizer states in GB (56 GB). The * 2 is because the Adam optimizer stores two values per parameter: the first moment (exponential moving average of gradients) and the second moment (exponential moving average of squared gradients).
• total_vram_needed = model_weights + gradients + optimizer_states — sums all three: 28 + 28 + 56 = 112 GB.
• # Adam optimizer stores 2 values per param — this comment explains the * 2 multiplier. If you used SGD instead of Adam, you’d only need the weights and gradients (56 GB), but Adam’s momentum tracking adds another 56 GB.

For Rae, this is why her consumer GPU — which handles RAG inference just fine at 28 GB for the weights — chokes the moment she tries to train on it: training needs 4× the memory of inference, not 2× or 1×.

In full fine-tuning, the weight update is applied directly to the full weight matrix:

$W_{\text{new}} = W + \Delta W$

where $\Delta W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ is a full-rank matrix with the same dimensions as $W$. LoRA replaces this with a low-rank approximation:

$\Delta W \approx A \times B$

where $A \in \mathbb{R}^{d_{\text{out}} \times r}$, $B \in \mathbb{R}^{r \times d_{\text{in}}}$, and $r \ll \min(d_{\text{in}}, d_{\text{out}})$. The forward pass becomes:

$h = W x + A B x$

The original weights $W$ are frozen (no gradients computed), and only $A$ and $B$ are trained. The total trainable parameters drop from $d_{\text{in}} \times d_{\text{out}}$ to $r \times (d_{\text{in}} + d_{\text{out}})$.

The key insight is that $r$ (the rank) is the bottleneck: it controls the tradeoff between expressiveness (how many different changes the adapter can represent) and memory (how many parameters need gradients). With $r = 8$ and $d_{\text{in}} = d_{\text{out}} = 4096$, the full matrix needs $4096 \times 4096 = 16{,}777{,}216$ parameters, but $A \times B$ needs only $4096 \times 8 + 8 \times 4096 = 65{,}536$ — a 256× reduction.

This block computes the parameter count for full fine-tuning vs. LoRA on a single 4096×4096 layer, demonstrating the dramatic reduction.

• import torch and import torch.nn as nn — import PyTorch and its neural network module. Neither is actually used in this calculation; they’re imported because this snippet would normally be part of a larger training script.
• in_dim, out_dim = 4096, 4096 — sets the input and output dimensions of a typical transformer layer (this matches the hidden size of models like Llama-7B).
• rank = 8 — sets the LoRA rank, the r hyperparameter that controls the bottleneck.
• full_params = in_dim * out_dim — computes the number of parameters for full fine-tuning: 4096 × 4096 = 16,777,216 (about 16.8M).
• matrix_a = in_dim * rank — computes the parameters in matrix $A$: 4096 × 8 = 32,768.
• matrix_b = rank * out_dim — computes the parameters in matrix $B$: 8 × 4096 = 32,768.
• lora_params = matrix_a + matrix_b — sums both: 32,768 + 32,768 = 65,536 total trainable parameters.
• reduction = full_params / lora_params — computes the ratio: 16,777,216 / 65,536 = 256.0.
• print(f"Full parameters: {full_params:,}") — uses the :, format specifier to add thousands separators for readability.
• print(f"Reduction factor: {reduction:.1f}x") — formats the reduction to one decimal place with a literal “x” suffix.

The output shows a 256× reduction — meaning we compute and store gradients for 65K parameters instead of 16.8M, which is what transforms “impossible on consumer hardware” to “trivial.” For Rae, this is why her GPU stops screaming: the gradients and optimizer states are now 256× smaller, fitting comfortably in her VRAM.

This block loads a pre-trained model and wraps it with LoRA adapters using the HuggingFace peft library.

• from peft import LoraConfig, get_peft_model — imports the configuration class and the wrapper function from the peft (Parameter-Efficient Fine-Tuning) library.
• from transformers import AutoModelForCausalLM — imports the auto-class for loading causal language models (models that predict the next token).
• model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m") — loads Facebook’s OPT-350M model, a relatively small model used here for demonstration (Rae would use a larger model like Llama-7B or Mistral-7B in practice).
• config = LoraConfig(...) — creates a LoRA configuration object with six parameters.
  • r=16 — sets the rank to 16. Higher rank means more expressive adapters but more memory; the comment explains this tradeoff.
  • lora_alpha=32 — a scaling factor that controls how strongly the adapter influences the output. The effective scaling is alpha / r, so 32/16 = 2.0 here.
  • target_modules=["q_proj", "v_proj"] — specifies which weight matrices in the transformer to attach adapters to. q_proj (query projection) and v_proj (value projection) are the attention layers, the standard targets for LoRA; you could also add k_proj, o_proj, or the MLP layers.
  • lora_dropout=0.05 — applies 5% dropout to the adapter outputs to prevent overfitting.
  • bias="none" — means no bias parameters are trained, keeping the adapter even smaller.
  • task_type="CAUSAL_LM" — tells the library this is a causal language modeling task (next-token prediction).
• lora_model = get_peft_model(model, config) — wraps the base model with the LoRA configuration. This freezes all original weights and injects the trainable adapter matrices into the target modules.
• lora_model.print_trainable_parameters() — prints a summary showing how many parameters are trainable vs. frozen.

The output trainable params: 1,572,864 || all params: 332,769,280 || trainable%: 0.4727 means only about 1.57M out of roughly 333M parameters are being trained — under half a percent. (OPT-350M has 24 transformer layers, each with a 1024×1024 q_proj and v_proj; at r=16 each targeted matrix adds 1024×16 + 16×1024 = 32,768 adapter parameters, so 24 layers × 2 modules × 32,768 = 1,572,864 — exactly the printed count.) For Rae, this means her GPU only needs to store gradients and optimizer states for ~1.57M parameters, not 333M — which is what makes the fine-tune feasible on her hardware.

This block configures 4-bit quantization for loading a model at a fraction of its normal memory cost — the key technique that makes QLoRA work on consumer GPUs.

• from transformers import BitsAndBytesConfig — imports the quantization configuration class from HuggingFace Transformers.
• quant_config = BitsAndBytesConfig(...) — creates a configuration object with four parameters that control how the model’s weights are compressed.
  • load_in_4bit=True — tells the loader to quantize all weights to 4 bits, reducing each parameter from 16 bits (2 bytes) or 32 bits (4 bytes) to just 0.5 bytes, a 4× to 8× reduction.
  • bnb_4bit_quant_type="nf4" — specifies the quantization type as NormalFloat 4 (NF4), a quantization scheme designed by the QLoRA authors specifically for normally-distributed weight values. It’s better than uniform quantization because it allocates more precision to the most common weight values.
  • bnb_4bit_compute_dtype=torch.float16 — specifies that during the forward pass, weights are dequantized back to float16 (16-bit) for computation. This is the “just-in-time dequantization” the article describes.
  • bnb_4bit_use_double_quant=True — enables double quantization, quantizing the quantization constants themselves, saving an additional ~0.4 bits per parameter.
• model_4bit = AutoModelForCausalLM.from_pretrained(...) — loads Mistral-7B with the quantization config applied.
  • quantization_config=quant_config — passes the 4-bit configuration.
  • device_map="auto" — lets HuggingFace automatically place model layers across available GPUs, essential when a model is too large for a single device but can be split.

For Rae, this is what lets her 7B model (which needs 28 GB in float32) fit in about 5 GB of VRAM — leaving plenty of room for the LoRA adapters and their gradients on her single consumer GPU.

This block assembles a complete training pipeline using the SFTTrainer (Supervised Fine-Tuning Trainer) from the trl library — the final piece that ties QLoRA loading, LoRA adapters, and training data together.

• lora_model = get_peft_model(model_4bit, config) — wraps the 4-bit quantized base model from Section 6 with the same LoraConfig from Section 4. This is the line that actually makes it QLoRA rather than plain LoRA: the base model is quantized and has trainable adapters attached.
• from trl import SFTTrainer, SFTConfig — imports the supervised fine-tuning trainer, which handles the training loop for instruction-tuning datasets, and its configuration class. (In current trl releases, dataset options like dataset_text_field and the sequence-length cap live on SFTConfig, not as direct keyword arguments to SFTTrainer — passing them straight to SFTTrainer raises TypeError: SFTTrainer.__init__() got an unexpected keyword argument 'dataset_text_field'.)
• trainer = SFTTrainer(...) — creates the trainer with two top-level arguments plus a config object.
  • model=lora_model — passes the LoRA-wrapped, 4-bit quantized model built above. This is what makes it QLoRA: the model is both quantized and has trainable adapters.
  • train_dataset=my_data — passes the training dataset — for Rae, this would be her hundreds of (question, answer) pairs from past support tickets.
  • args=SFTConfig(...) — bundles both the dataset options and the standard training hyperparameters into one config object.
    • dataset_text_field="text" — tells the trainer which field in the dataset contains the text to train on.
    • max_length=512 — caps the sequence length to 512 tokens — shorter sequences use less VRAM, which matters on consumer GPUs; Rae’s support-bot Q&A pairs are short enough to fit comfortably. (This option was called max_seq_length in older trl releases; it was renamed max_length when the dataset options moved onto SFTConfig.)
    • per_device_train_batch_size=1 — sets the batch size to 1, the most memory-conservative setting, essential for small GPUs.
    • gradient_accumulation_steps=4 — accumulates gradients over 4 steps before updating weights. This simulates a batch size of 4 (1 × 4) without needing the memory for 4 simultaneous examples.
    • learning_rate=2e-4 — sets a learning rate of 0.0002, note this is higher than the 2e-5 used in full fine-tuning (covered in the prior article’s conceptual example), because LoRA adapters need a stronger nudge to learn effectively with fewer parameters.
    • logging_steps=10 — logs training metrics every 10 steps.
    • output_dir="./outputs" — specifies where to save checkpoints.
• trainer.train() — starts the training loop.

For Rae, this is the exact pipeline she runs on her single GPU: load Mistral-7B in 4-bit, attach LoRA adapters, feed in her support-ticket Q&A pairs, and train — the whole thing fits in under 8GB of VRAM.

Full Fine-Tuning vs. LoRA vs. QLoRA: Which Should Rae Pick?

Rae’s startup has specific constraints that narrow the field to one option: a single consumer GPU (let’s say 8–12GB VRAM), a small budget (no cloud GPU rentals), and a few hundred high-quality training examples (her support-ticket resolutions). Here’s how the three approaches stack up:

The decision for Rae’s startup:

Rae has a single consumer GPU with 8–12 GB of VRAM and a few hundred training examples. QLoRA is her only realistic option. It lets her fine-tune a 7B model (which she needs for quality) on hardware she already owns, without renting cloud GPUs by the hour. The 4-bit base model occupies ~5 GB, and the LoRA adapter gradients add only a few hundred MB — the whole pipeline fits in her VRAM with room to spare.

When NOT to use QLoRA (and what to use instead):

• If Rae had 24 GB+ VRAM (e.g., an RTX 4090): LoRA without quantization would be preferable — it’s slightly faster (no dequantization overhead) and avoids any quantization noise in the base model. The quality difference is small but measurable on benchmarks.
• If Rae had millions of examples and a cluster of H100s: Full fine-tuning could squeeze out additional quality — the full-rank weight update can represent changes that a rank-8 or rank-16 adapter cannot. But this is almost never the case for startups, and the marginal quality gain rarely justifies the 10×+ cost.
• If Rae needed maximum quality and had 24 GB VRAM but not millions of examples: LoRA with a higher rank (e.g., r=64 or r=128) would give her more adapter capacity without needing full fine-tuning — the rank controls how expressive the adapter can be, and bumping it up is cheaper than going to full fine-tuning.

The key tradeoff to understand: QLoRA adds quantization noise to the base model — the 4-bit weights are slightly less precise than their 16-bit originals. In practice, the LoRA adapters compensate for this noise during training, and the quality gap between QLoRA and full fine-tuning is small enough that it rarely matters for production use cases like support bots. The tradeoff is: accept a tiny quality hit in exchange for fitting on hardware you already own.

This block merges the trained LoRA adapters back into the base model’s weights, producing a single self-contained model with no inference overhead.

• merged_model = lora_model.merge_and_unload() — performs the merge: it multiplies the trained $A$ and $B$ matrices ($A \times B = \Delta W$), adds the result to the frozen base weights ($W + \Delta W$), and then removes the adapter layers from the model. The merge_and_unload() method does both in one call. After merging, the model behaves exactly like a standard fine-tuned model: the low-rank approximation is baked into the weights, and there’s no separate adapter to load at inference time.
• merged_model.save_pretrained("my-final-model") — saves the merged model to disk as a standard HuggingFace checkpoint, a single directory containing the model weights and config that can be loaded with AutoModelForCausalLM.from_pretrained("my-final-model") without any LoRA-specific setup.

For Rae, this is the final step: she trains her QLoRA adapters on her support-ticket data, merges them into the base model, saves the result, and deploys the merged model alongside her RAG pipeline — at inference time, there’s no adapter overhead, just a model that happens to speak in her company’s support-team voice.