Fine-Tune Local LLMs with LoRA & Unsloth [Python Guide]

Zenon J September 08, 2026
Machine Learning LLM Fine Tuning Weights Unsloth LoRA Python GPU

While prompt engineering and Retrieval-Augmented Generation (RAG) handle factual knowledge injection, teaching a foundation model specialized reasoning patterns, strict enterprise JSON formats, domain-specific coding styles, or non-English dialects requires Fine-Tuning.

Traditional full parameter fine-tuning of an 8B model requires multiple enterprise GPUs with over 64 GB of VRAM. With Unsloth and QLoRA (Quantized Low-Rank Adaptation), fine-tuning modern open-source models (such as Llama 3.1 or Mistral) can be executed on a single consumer GPU with up to 80% less memory and 2x to 5x faster speeds.


Full Fine-Tuning vs. LoRA vs. QLoRA Comparison


Prerequisites & Installation

Unsloth requires an NVIDIA GPU with CUDA 11.8 or 12.1+ support and PyTorch 2.1+.

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps "xformers<0.0.27" trl peft accelerate bitsandbytes
Fine-Tuning MethodTrained ParametersVRAM Footprint (8B Model)Training SpeedAdapter Size
Full Parameter Fine-Tuning100% of all model weights> 64 GB VRAM (Multi-GPU)Baseline (1x)Full model weights (~16 GB)
Standard LoRA~0.5% low-rank adapter matrices~16 GB VRAMModerate (1.5x)Small adapter (~100 MB)
Unsloth QLoRA4-bit quantized base + LoRA< 8 GB VRAM (Single GPU)Ultra-Fast (2x - 5x with custom CUDA kernels)Compact adapter (~50 MB)

Understanding Low-Rank Adaptation (LoRA) Mechanics

During full fine-tuning, the optimizer computes gradients for billions of matrix parameters. LoRA freezes the pretrained foundation model weights and decomposes the weight update matrix (delta W) into two low-rank matrices (A and B).

This drastically reduces the number of trainable parameters from 8 billion to just ~20 million, allowing training on standard desktop consumer graphics cards.


High-Speed Fine-Tuning Pipeline with Unsloth in Python

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048
load_in_4bit = True

# 1. Load Pretrained 4-bit Quantized Base Model
print("Loading 4-bit base model via Unsloth...")
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b-bnb-4bit",
    max_seq_length=max_seq_length,
    load_in_4bit=load_in_4bit,
)

# 2. Inject LoRA Low-Rank Adapters
print("Injecting LoRA adapters into attention & MLP projection layers...")
model = FastLanguageModel.get_peft_model(
    model,
    r=16,               # LoRA Rank
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_alpha=16,
    lora_dropout=0,     # Optimized 0 dropout for Unsloth
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

print("Model successfully configured with LoRA adapters ready for dataset training.")

Dataset Preparation & Quality Standards

  • Quality over Quantity: 1,000 high-quality, verified instruction-response pairs produce significantly better fine-tuned models than 50,000 noisy, machine-translated examples.
  • Chat Template Formatting: Always apply the model's official tokenizer chat template (such as Llama-3 or ChatML) to format system, user, and assistant turns consistently.
  • Learning Rate Warmup: Use a cosine learning rate scheduler with a warmup ratio of 0.05 to prevent catastrophic forgetting during initial training steps.

Frequently Asked Questions

Q: Can I export fine-tuned Unsloth models to Ollama or GGUF?
A: Yes. Unsloth supports one-line export to GGUF format via model.save_pretrained_gguf('model_name', tokenizer, quantization_method='q4_k_m'), which can be loaded directly into Ollama.

Q: Does LoRA degrade the model's general knowledge?
A: When trained with low learning rates (e.g. 2e-4) and proper rank sizing, LoRA adapts the model's behavior while preserving the broad knowledge base of the foundation model.

Post a Comment

0 Comments