Open-weight AI models — where the trained weights are publicly released — represent a different value proposition from closed, API-only services like Claude or GPT. They offer customisability, data sovereignty, and fixed infrastructure costs at the expense of raw capability and developer experience. This article covers the major model families, how to train or adapt them for specific tasks, and how to deploy them at various scales.
The Open-Weight Model Landscape
The stack has four layers, each with distinct options:
- The model — the actual trained weights and architecture (Llama, Mistral, Qwen, DeepSeek, Gemma, Phi)
- The format — how weights are saved and compressed (GGUF, AWQ, GPTQ, Safetensors)
- The engine — the runtime that executes inference (llama.cpp, vLLM, TensorRT-LLM, SGLang)
- The orchestrator — the UI or tool wrapping the engine (Ollama, LM Studio, AnythingLLM)
Log scale — the gap between a 3.8B edge model and a 671B frontier model spans two orders of magnitude.
Meta: Llama
Meta's Llama family — available at 8B, 70B, and 405B parameters — is the de facto baseline for open-weight research. The architecture is a dense, decoder-only transformer with grouped-query attention, extended context windows (up to 128K tokens), and stable training dynamics. Meta releases models under a permissive commercial licence, which means native integration across every major inference runtime. Llama 3.1 405B is the first open-weight model to approach GPT-4 on standard benchmarks. Its primary weakness is that it requires significant DevOps investment — there is no hosted interface; you provision and operate the infrastructure yourself.
Mistral and the Mixture-of-Experts Architecture
Mistral AI pioneered sparse Mixture-of-Experts (MoE) in the open-weight space. In a dense model, every parameter is activated for every token. In an MoE model, a learned router assigns each token to a small subset of specialist networks ("experts"), leaving the rest idle. Mixtral 8×7B has approximately 47B total parameters but activates only ~13B per token — delivering the intelligence of a much larger model at the compute cost of a smaller one.
Dense Model
Every parameter activates for every token
MoE (Mixtral 8×7B)
A router activates only 2 of 8 experts per token
~47B total parameters, only ~13B active per token — the intelligence of a larger model at the compute cost of a smaller one.
Mistral models are also notable for their European data infrastructure and GDPR-aligned deployment options, making them particularly relevant for organisations in regulated EU contexts.
Other Notable Families
- Qwen (Alibaba) — strong multilingual capability and long-context support (up to 1M tokens in Qwen2.5). Commercially deployable but subject to geopolitical friction in Western markets.
- DeepSeek — highly capable MoE architecture (671B total, ~37B active in V3), released at very low inference cost. Political content restrictions apply; unsuitable for general-purpose deployment.
- Gemma (Google) — compact models (2B, 7B, 9B, 27B) optimised for on-device and edge inference. Instruction-tuned variants available.
- Phi (Microsoft) — Small Language Models optimised for reasoning-dense tasks on constrained hardware. Phi-3 Mini runs on a mobile phone.
Training Methods
If you want to teach a model new knowledge, a new language, or a new behaviour, there are three strategies — each with different data, compute, and risk profiles.
- Teaches new domain knowledge
- Massive raw text corpora
- Days–weeks on H100 clusters
- Teaches behaviour, not knowledge
- Thousands of curated examples
- Hours–days on one GPU
- Small adapter matrices only
- 99%+ fewer trainable params
- Runs on a single GPU
Continued Pre-Training (CPT)
Unsupervised training on massive raw text corpora. Used when the base model lacks the vocabulary, concepts, or domain knowledge you need — for example, training on a corpus of Irish legislation, public sector procurement documents, or proprietary engineering specifications. Compute requirements are high: H100 clusters running for days or weeks. This is the right tool when the model fundamentally does not know the domain.
Supervised Fine-Tuning (SFT)
Training on curated prompt–response pairs to change the model's behaviour rather than its knowledge. The model already understands the domain; you are teaching it how to respond — always output structured JSON, adopt a specific voice, follow a particular workflow. Requires thousands of high-quality examples. Compute is moderate: an enterprise GPU (A100, H100) running for hours to days.
Parameter-Efficient Fine-Tuning: LoRA and QLoRA
The most practical entry point for most organisations. Rather than retraining the entire model, LoRA (Low-Rank Adaptation) freezes the original weights and trains only a pair of small adapter matrices inserted into each transformer layer. The total number of trainable parameters drops by 99%+ compared to full fine-tuning, with minimal capability loss.
B×A is always low-rank — it captures only the task-specific adjustment, never the full weight space.
QLoRA extends this by quantising the frozen base model to 4-bit integers, reducing VRAM requirements by approximately 75%. A 70B model that would normally require eight 80GB A100s can be fine-tuned on a single A100 with QLoRA. At inference, the low-rank matrices are merged back into the base weights — zero latency penalty in production.
Where W is the frozen original weight, B and A are the low-rank adapter matrices, r is the rank (typically 4–64), and α is a scaling factor. The product BA is always low-rank — it captures only the task-specific adjustment, not the full weight space.
Deployment
Deployment architecture is determined by three variables: traffic volume, data sensitivity, and the organisation's infrastructure capability. The decision should be made before choosing a model — the right model for a local deployment is a different choice from the right model for a production vLLM cluster.
- 7B–13B, GGUF quantised
- Consumer GPU or Apple Silicon
- Individuals, experimentation
- PagedAttention, continuous batching
- A100 / H100 instances
- Sustained production traffic
- Zero infrastructure to operate
- Usage-based pricing
- Regulated-sector default
Local and Desktop Deployment
Ollama and LM Studio abstract all inference complexity into a single install. Both use llama.cpp under the hood, which splits model weights between CPU RAM and GPU VRAM using GGUF quantisation — making 7B to 13B models practical on Apple Silicon MacBooks and consumer NVIDIA cards (RTX 3090+). This tier is appropriate for individual developers, privacy-sensitive local workflows, and experimentation. It does not scale.
Production Deployment
vLLM is the standard production inference engine for open-weight models. Its PagedAttention algorithm manages GPU memory as efficiently as a database page cache, and continuous batching ensures GPUs never idle waiting for individual requests. TGI (Hugging Face Text Generation Inference) and SGLang are alternatives with different throughput/latency trade-offs. Production deployments run on A100 or H100 instances via AWS EC2, GCP, RunPod, or Lambda Labs.
Cost Efficiency
Two techniques materially reduce inference cost without degrading output quality.
Accepted tokens ship immediately; rejected ones are regenerated. Net result: 2–3× faster generation at the same quality.
Speculative decoding uses a small draft model (e.g. a 7B) to rapidly generate K candidate tokens, then passes the candidates to the large target model (e.g. 70B) for parallel verification in a single forward pass. Tokens the target model accepts are output immediately; rejected tokens are regenerated. Net result: 2–3× faster generation at the same quality level as running the large model auto-regressively.
Prompt caching is effective when every request shares a long system prompt or retrieval context. Inference engines like vLLM retain the KV cache for the shared prefix across requests — subsequent prompts only process the new user query, not the entire context. On workloads with 2,000-token system prompts, this can cut token processing by 60–80%.