Local LLMs 2 years later: what changed, what’s obsolete, what to run instead in 2026

Two years ago I wrote about running Llama 3 8B (released 2024–04–18) locally on CPU by hand-patching Meta’s official repo: ripping out .cuda()calls, faking a single-node torch.distributed group just to get one model to answer a prompt, waiting 90 seconds for four completions. It worked, and it felt like a small victory.

Press enter or click to view image in full size

Today the equivalent is:

ollama run qwen3.6   # Qwen3.6, released 2026-Apr

No forked repo, no torch.distributed workaround, no hunting for hardcoded .cuda() calls. I went back through Ollama's and Hugging Face's own blogs from this spring and summer to see what actually changed, rather than trust my own memory of the space. Here's what's in the primary sources, with links and release dates for every model mentioned.

Before Ollama became the default local inference layer

Worth remembering what “running a model locally” meant before Ollama ate this category. In 2024-Apr, when Llama 3 8B shipped, running it meant: cloning Meta’s reference repo, installing the exact PyTorch/CUDA versions it was pinned to, and hoping your GPU (or lack of one) matched what the code assumed. There was no standard packaging format for weights across projects: you juggled raw PyTorch checkpoints, GGML, then GGUF, each with different loaders. Every model came with its own inference script, its own quirks, its own .cuda() calls baked in for people who didn't have CUDA. Getting a second model running usually meant repeating the whole process from scratch in a different repo with a different set of assumptions.

Ollama collapsed all of that into one runtime with one weight format (GGUF) and one command. That’s the shift underneath everything below.

Ollama became the default local inference layer

Ollama’s own posts this year read less like “here’s a CLI update” and more like “here’s the infrastructure layer for open models”:

• Ollama’s new engine for multimodal models (2025–05–15): Ollama moved off relying solely on llama.cpp and built a dedicated multimodal engine on the GGML tensor library. The launch line-up was Llama 4 (2025-04-05), Gemma 3 (2025-03-12), Qwen2.5-VL (2025-01-28), and Mistral Small 3.1 (2025-03-17). Each model ships its own projection layer, so there's no more per-model glue code.
• Ollama is now powered by MLX on Apple Silicon in preview (2026–03–30): a preview build using Apple’s MLX framework instead of pure llama.cpp/Metal, benchmarked on M5 chips running Qwen3.5–35B. Prefill jumped from 1,154 to 1,810 tokens/sec, decode from 58 to 112 tokens/sec. It also adopts NVIDIA’s NVFP4 quantisation format to cut memory bandwidth without giving up accuracy. Requires 32GB+ unified memory, wich any M-processor has since 2022.
• Ollama’s new app: a proper desktop app for macOS/Windows/Linux with drag-and-drop files/PDFs, adjustable context length, and image input for multimodal models.
• Ollama: all aboard open models (2026–07–09): announcing an $88M raise, and more relevant to this article, Ollama’s cloud tier now fronts the largest open models people actually can’t run locally (GLM, Nemotron, DeepSeek, Kimi, MiniMax) through the same ollama run interface.

The practical effect for you: pick a model, ollama pull or ollama run it, and the runtime figures out quantisation, multimodal handling, and (if the model's too big) whether to hand it off to cloud. That's what the Docker for containers, to model weights.

What Hugging Face’s own data says about the ecosystem

Hugging Face’s State of Open Source on Hugging Face: Spring 2026 (2026–03–17) is the most useful primary source here because it’s their own usage data, not vibes:

• Qwen is the most derived-from family on the Hub: over 113,000 direct derivative models, 200,000+ counting tagged variants. That’s community adoption you can measure, not marketing.
• DeepSeek passed Llama as the single most-liked model family on the Hub, sustained across V3, R1, and V3.2 releases rather than one lucky launch.
• Llama went from “the” open model to one of several serious options in an increasingly international field, with the US, China, and Europe all shipping competitive open weights now.
• Their own framing, worth quoting directly because it answers your question about non-coding usefulness: “models with hundreds of millions of parameters support search, tagging, and document processing workflows, while models in the single-digit billions are widely used for coding, reasoning, and multimodal tasks.” Coding is one bucket among several, not the main event.

Why release date matters more than the model name

Once you start attaching dates to these models, a pattern jumps out. DeepSeek V3 (2024–12–26) led to R1 (2025–01–20), then V3.2 (2025–12–02), then V4-Flash (2026–07–31). Qwen2.5-VL (2025-Jan) led to Qwen3.5 (2026–02–16), then Qwen3.6 (2026-Apr). Gemma 3 (2025-Mar) led to Gemma 3n (2025-May), then Gemma 3 270M (2025–08–14). These labs aren’t shipping once a year, they’re shipping every 1–4 months, and each release usually folds in a new training recipe, more/cleaner training data, better post-training (RLHF/RLVR), and architecture tweaks (MoE routing, attention variants, quantisation-aware training) that the previous release didn’t have.

That’s why “a couple of months newer” is a real performance signal, not just a version-number bump:

• Post-training methods move fast. The gap between a base model and a well-post-trained one is often bigger than the gap between two base model generations. A model fine-tuned with a 2026-Sep RL recipe will out-reason a same-size model from six months earlier even if the underlying architecture is identical.
• Distillation compounds. Newer small models are frequently distilled from newer, stronger frontier models (Gemma 3 from Gemini 2.0, for instance), so a small model’s ceiling rises every time the frontier model it’s distilled from improves, independent of the small model’s own architecture changing at all.
• Benchmarks get gamed, then re-solved. Each generation closes gaps exposed by the previous one’s benchmark failures (tool-calling reliability, long-context degradation, multilingual accuracy) because labs iterate directly against where the last release was criticised.
• Quantisation and serving techniques improve independently of the weights. Ollama’s own MLX/NVFP4 work above is a case in point: the same weights, released months apart from a serving-technique standpoint, run meaningfully faster and use less memory purely because the runtime caught up.

The corollary: anything older than about a year in this space is most likely obsolete, not because it stopped working, but because a same-size or smaller successor almost certainly beats it on quality, speed, and efficiency simultaneously. Llama 3 8B (2024–04–18) was state-of-the-art in its class at launch. By the time Llama 4 shipped a year later, let alone Qwen3.5 or Gemma 3n, it was a teaching example, not a production choice. If a model card doesn’t show meaningful activity (updates, derivatives, a named successor) in the trailing 12 months, treat it as a historical reference, not a shortlist candidate. Check the model’s Hugging Face page for its release date and compare it against whatever’s newest in the same size class before you commit to it.

Models like Qwen, Gemma, and others for everyday (non-coding) work. Gemini ≠ Gemma

Worth clearing up first: Gemini and Gemma are not the same thing. Gemini (2.x/3.x) is Google’s closed, hosted frontier model. You call it via API, you don’t run it on your laptop. Gemma is Google’s open-weight sibling, distilled from the same research, and that’s the one that shows up in Ollama and on Hugging Face. If you type “gemini” into ollama run, you'll get nothing. You want gemma.

With that sorted, here’s why these general-purpose open models are genuinely useful for the non-coding majority of what people actually do with an LLM: writing, summarising, translating, answering questions, light research, voice/chat assistants.

• Gemma 3 (2025–03–12) / Gemma 3n (2025-May) / Gemma 3 270M (2025–08–14): Google’s own developer blog is explicit that this line is built for exactly this. The Gemma 3n announcement is a mobile-first, multimodal (audio+image+text) model aimed at on-device transcription, translation, and assistant tasks, not code. The Gemma 3 270M post reports it used 0.75% battery for 25 conversations on a Pixel 9 Pro. This is the tier for “always-on, runs on a phone, does classification/summarisation all day without you noticing.”
• Qwen3.5 (2026–02–16) / Qwen3.6 (2026-Apr): Alibaba’s newer generation spans 0.8B up to 397B-A17B (a mixture-of-experts model), and per its own model cards on Hugging Face, it’s built around switchable “thinking” vs “non-thinking” modes. Thinking mode handles reasoning and maths, non-thinking mode gives fast, low-latency conversational replies, which is exactly the mode you want for everyday chat, drafting, and Q&A rather than paying reasoning-latency tax on every message. Qwen3.6 replaced it two months later.
• GLM 5.2 (2026-Jun, mid-month), Kimi K3 (2026–07–16), MiniMax H3/Hailuo 3.0 (2026–07–31), DeepSeek V3 (2024–12–26) through R1 (2025–01–20) to V3.2 (2025–12–02) to V4-Flash-0731 (2026–07–31), and Nemotron 3 Nano/Super/Ultra (2025–12–15 / 2026–03–11 / 2026–06–04): these are the models Ollama’s cloud tier fronts because they’re too large for a laptop but are strong general reasoning/chat/agentic models. DeepSeek in particular being the Hub’s most-liked family isn’t a coding story: people use it for general Q&A and reasoning at a price/performance point closed models struggle to match. Notice how tight the cadence is: three DeepSeek releases in under 13 months, three Nemotron tiers in under six.
• gpt-oss (2025–08–05): OpenAI’s open-weight release now runs directly in Ollama’s library (gpt-oss:20b fits on a 16GB card per community benchmarks referencing Ollama's own model listings), giving a non-coding-focused reasoning/agent option from a lab that historically only shipped closed models.

The common thread across all of these primary-source posts: the interesting story in 2026 isn’t “which model writes the best code.” It’s that a 1B-14B general model now comfortably handles the bulk of everyday knowledge work (drafting, summarising, translating, answering, light multimodal understanding) locally, privately, and fast enough that you stop thinking about it as “running a model” and start thinking of it as an app.

Matching a model to your hardware, straight from Hugging Face

The other primary source worth citing directly: Hugging Face’s Filter Models page by Hardware changelog entry (2026–06–30). Instead of guessing whether a model fits your machine, you tell the Hub what you own (a specific GPU, a CPU, or an Apple Silicon chip) in your Hardware settings, and the Models page filters to only what will actually run there. It stacks with existing filters and is shareable via URL, so a link like huggingface.co/models?apps=llama.cpp&hardware=apple-m4-max points anyone (even logged out) at the llama.cpp models that fit an Apple M4 Max. The compatibility data comes from what the community actually owns: over 300,000 members have shared their hardware specs to build it. Model pages with GGUF or MLX files now also show a Hardware compatibility panel estimating whether each quantisation runs on your saved hardware before you download a single byte.

If you’d rather do the maths yourself, the rule of thumb hasn’t changed: roughly 8GB of VRAM/unified memory runs a 7B model well, 12GB covers 14B, 16GB covers ~24B, 24GB covers 32B, and 70B-class models want 40GB+ for a usable Q4_K_M quant. Every GGUF repo lists the exact file size per quant level in its Files tab, and that file size is your real requirement, no calculator needed.

Final thought

In 2024 I spent a day fighting torch.distributed and hardcoded CUDA calls just to get one 8B model talking on CPU. In 2026 I type one line and get the job done. The frontier keeps moving, but according to the people building the tools, the floor for "useful model, running on my own machine, for things that aren't code" dropped further and faster than almost anyone predicted two years ago.

Installing Ollama itself takes less effort than reading this article: a native app for macOS, Linux, Windows — all from ollama.com. No account, no API key, no data leaving your machine: the model runs against your own hardware, so nothing you type or paste ever reaches a server. And “your own hardware” doesn’t have to mean a workstation. Even a base MacBook Air, with its unified memory and no fan to speak of, comfortably runs a 7–8B model at Q4 for everyday chat, drafting, and summarising. Pick a model sized to what you’ve got, per the hardware notes above, and you get real privacy.