Local LLMs: Choosing a Runtime and Estimating Memory

Local inference can provide offline operation and control over model files. It also transfers responsibility for memory, updates and access control to the application owner. The right starting point is a bounded task and hardware budget, not a promise that any laptop can run a large model well.

Estimate weights before runtime overhead

For a dense model with 7 billion parameters, an idealized four-bit weight representation needs:

7,000,000,000 × 4 ÷ 8 = 3,500,000,000 bytes, or about 3.26 GiB.

For 70 billion parameters, the same calculation gives 35 billion bytes, about 32.6 GiB. These are weight-only estimates. Quantization metadata, unquantized tensors, the KV cache and working buffers add memory. Longer contexts and concurrent requests can increase the runtime requirement.

A file fitting in storage does not establish that it will fit in GPU memory or respond at an acceptable speed.

Choose a compatible runtime

llama.cpp supports CPU, GPU and hybrid execution through several backends, including Metal, CUDA and Vulkan. It is not a CPU-only runtime. Support still depends on the model architecture, build and hardware.

Ollama offers model management and serving. Check the exact available model tag rather than inventing a size suffix. Its FAQ also distinguishes local execution from cloud features and documents a local-only setting.

An OpenAI-compatible endpoint can simplify integration, but test the actual methods, streaming, tool behavior and error responses your application uses.

Measure cold and warm behavior

Record hardware, driver, runtime version, model file, quantization, context length and concurrency. Measure model-loading time separately from first-token latency and generation throughput.

Test both short and long inputs. Check whether the system swaps memory, becomes unresponsive or evicts other applications. A model that is usable alone may not be usable while an editor, browser and build process are running.

Compare accepted answers for the intended task, not only tokens per second.

Verify the data boundary

Local model execution does not make every surrounding feature local. A UI may call analytics, remote search, cloud fallback or an embedding service. Review configuration and network behavior before claiming that sensitive input never leaves the device.

Protect any exposed inference endpoint and decide how prompts, outputs and logs are retained. Offline use also requires the necessary model files and assets to be present.

Local inference has hardware, energy and maintenance costs. Choose it when those costs and responsibilities fit the workflow; do not treat the absence of per-request API billing as zero total cost.

Sardar Mukhshaf

Full-Stack & DevOps Engineer

I build scalable web applications, secure backend systems, and production-ready cloud infrastructure. My engineering focus spans interface design, API architecture, deployment pipelines, and observability. Follow for more deep dives into system architecture, performance engineering, and emerging technologies.

ShareTwitterLinkedInCopy Link