How to Run Large Language Models Locally

Running a language model on your own computer means the model weights live on your disk, the computation happens on your processor or graphics chip, and no prompt leaves the machine. This has real benefits for privacy, offline work and cost, but it also brings limits. Memory size, not raw speed, usually decides which models are possible.

This post explains the main constraints, the role of quantization, and the kinds of tools people use to get started.

What local means in practice

A language model is a large set of numbers called weights. To generate text, the software multiplies these weights with the current input many times per token. Every weight must be available in fast memory while the model runs. If the weights do not fit, the software either refuses to load the model or swaps data to disk, which is far too slow for interactive use.

So the first question is not how fast a computer is but how much memory it has and how much of that memory the model can use. On Apple silicon, the CPU and GPU share one pool of unified memory, which lets a large share of system RAM serve the model. On a typical Windows or Linux PC, the graphics card's own video memory is the tighter limit.

Parameters, bytes and memory

A model's size is usually quoted in parameters, such as 7 billion or 70 billion. Each parameter stored at 16-bit precision takes 2 bytes. A 7 billion parameter model therefore needs about 14 GB just for weights at 16-bit precision. A 70 billion parameter model needs about 140 GB. Those numbers are too large for most personal machines.

Extra memory is also needed for the key-value cache, which stores attention data for the conversation so far, and for the runtime itself. Long conversations grow this cache, so a model that barely fits on an empty chat can fail after many turns.

Quantization makes models smaller

Quantization stores each weight with fewer bits. Common formats use 8 bits, 6 bits or 4 bits per weight instead of 16. Going from 16 bits to 4 bits cuts weight memory by roughly four times, at the cost of some accuracy. Good quantization schemes choose scale factors per small block of weights so that the rounding error stays small in the places that matter most.

In practice, a well-made 4-bit version of a model usually keeps most of its everyday quality, while very aggressive quantization below 4 bits can noticeably hurt reasoning and coding. Quality also depends on the task, so testing the model on your own work is the only reliable check.

Approximate memory needs

The table below gives rough weight-only figures. Real usage is higher because of the cache and overhead.

Model size	16-bit weights	4-bit weights	Typical fit
1 to 3 billion	2 to 6 GB	1 to 2 GB	Most laptops and phones with enough RAM
7 to 8 billion	about 14 to 16 GB	about 4 to 5 GB	Machines with 8 to 16 GB of memory
13 billion	about 26 GB	about 7 to 8 GB	Machines with 16 GB or more
70 billion	about 140 GB	about 35 to 40 GB	Workstations or high-memory machines

These are estimates from parameter count and bit width. Check the actual file size of any model you download before deciding.

Speed depends on memory bandwidth

Generating each token requires reading most of the weights from memory. As a result, token speed is often limited by memory bandwidth more than by raw compute. A machine with fast memory and a capable GPU will generate text faster than one with slower memory, even if the processors are similar. Prompt processing, the step where the model reads your input, is more compute-heavy and can be slow on large inputs.

This is why a quantized model that fits fully in fast memory usually feels much more responsive than a larger model that spills into slower storage.

Tools people commonly start with

Several open tools make local models practical. llama.cpp is a C and C++ inference engine that runs models stored in the GGUF format on CPUs, GPUs and Apple silicon. Ollama wraps similar engines in a simple command-line tool and local API. LM Studio offers a graphical app for browsing, downloading and chatting with models. On Apple silicon, the MLX framework is designed around unified memory and is another common choice.

Most of these tools can expose a local server that follows a widely used chat API shape. That lets other apps send requests to your own machine instead of a cloud provider, which is useful for development and testing.

Choosing a first model

Start with the largest model that fits comfortably with room left for the system and the conversation cache. A good rule is to keep at least a few gigabytes free so the machine stays responsive. Pick an instruction-tuned model, not a base model, for chat. For coding or long documents, check the context length, which sets how much text the model can consider at once.

Test two or three models on a task you care about. Public benchmarks are useful for rough ranking, but they do not always match your workload.

Context length and memory growth

Context length is the amount of text a model can consider at once, measured in tokens. Models advertise maximum context sizes, but using the full window costs memory. The key-value cache grows with every token in the conversation, so a long document can add gigabytes on top of the weights. Many local tools let you cap the context to a smaller value, which keeps memory use predictable at the cost of shorter conversations.

A practical habit is to start new chats for new tasks and paste in only the relevant excerpts. This keeps the cache small, speeds up replies and reduces the chance that the model loses track of details in a very long input.

Tradeoffs to expect

Local models give privacy and control, because data does not leave the machine, and they work offline. They also avoid per-token fees after the hardware is paid for. The costs are setup time, limited memory, lower speed on modest hardware, and usually weaker results than the largest cloud models on the hardest tasks. Updates arrive when you download them, not automatically.

Security still matters. Download models and tools from sources you trust, and check checksums when they are published. A model file is data, but the software that loads it runs code on your machine.

Key takeaways

Running a language model locally depends mainly on memory. Weight size is roughly parameter count times bytes per weight, and the cache and runtime add more on top. Quantization shrinks models by storing weights in fewer bits, and 4-bit versions are a common sweet spot for personal machines.

Choose the largest instruction-tuned model that fits with room to spare, test it on your real tasks, and use a tool such as llama.cpp, Ollama, LM Studio or MLX to run it. The result is private, offline and under your control, with tradeoffs in speed and capability that depend on your hardware.