How to Run LLMs on Your Computer

Tags: AI, AI Series, ChatGPT, GenAi, LLM

Quick Links: Resources for Learning AI | Keep up with AI | List of AI Tools

Disclaimer: I work for Dell Technology Services as a Workforce Transformation Solutions Principal. It is my passion to help guide organizations through the current technology transition specifically as it relates to Workforce Transformation. Visit Dell Technologies site for more information. Opinions are my own and not the views of my employer.

Large Language Models (LLMs) have revolutionized the field of natural language processing and artificial intelligence. These powerful models enable applications like language translation, text summarization, and content generation. The world of Large Language Models (LLMs) can be intimidating, especially with the associated costs. However, there are plenty of free and low-cost options available for those looking to dip their toes into this exciting field. There are many ways to explore, run and create AI applications on the cloud, there are many benefits of installing LLMs locally on your computer as well…

If you want to skip to the list of tools and links below click here

Benefits of Installing LLMs Locally

Category	Description
Privacy & Security	Your data never leaves your device Perfect for sensitive personal or business information No need to worry about cloud service privacy policies
No Internet Required	Work offline without interruption Ideal for travel or areas with poor connectivity Consistent performance regardless of internet speed
Cost-Effective	No subscription fees or API costs One-time setup with no recurring charges Unlimited usage within your hardware constraints Experimenting without the cost
Complete Control	Customize the model to your specific needs Fine-tune for specialized tasks No content filters or usage restrictions Customizable workflows Experiment with different models
Reduced Latency	Instant responses without network delays Smoother conversation flow Better integration with local applications

Some of the Use Cases to installing and running LLMs Locally:

Here’s your content formatted into a two-column table:

Personal	Professional
Writing assistant for offline work	Sensitive document analysis
Personal coding companion	Local code review and debugging
Local chatbot for learning and study	Customer data processing
Personal knowledge base management	Healthcare documentation assistance
Creative writing partner	Legal document analysis

Some more generic use cases include:
Proof of concept development
Text summarization: Generate concise summaries of long documents or articles.
Language translation: Translate text from one language to another with high accuracy.
Content generation: Write articles, blog posts, or even entire books using LLMs as your writing partner.
Chatbot development: Utilize LLMs to power conversational AI systems.
Data analysis: Process and analyze large datasets with the help of LLMs.

How Do Local LLMs Work?

Model Selection: Choose an LLM that suits your needs. There are various open-source models available, such as GPT-Neo, GPT-J, and LLaMA, that can be downloaded and run locally.
Installation: Set up the necessary software environment. This typically includes installing libraries and frameworks such as Python, PyTorch, TensorFlow, or other specialized tools like Hugging Face Transformers or Ollama, AnytimeLLM and other such tools (see list below) .
Hardware Requirements: Depending on the size of the model and your intended use, you may need a powerful machine with sufficient RAM and a capable GPU. Many models can be resource-intensive, so ensure your hardware can handle the demands.
Loading the Model: Once the environment is set up, load the model into memory. This involves downloading the pre-trained weights and configuration files.
Inference and Fine-Tuning: After loading, you can run inference tasks (like text generation or question answering) directly on your machine. Additionally, many frameworks allow you to fine-tune the model on specific datasets to better fit your needs.
User Interface: For ease of use, some applications provide user interfaces or APIs to interact with the model, making it simpler to integrate into applications or workflows (examples: AnytimeLLM, LM Studio)

How much hardware do I need to run LLMs locally?

Tools like Ollama, Anytime LLM, LM Studio make it SUPER easy to locally run LLMs. These kinds of tools, combined with a technique called LLM Quantization (dramatic reduction in model size, decreased memory usage, and improved inference speed)

The hardware requirements are MINIMAL: I have a small $250 Mini-PC (Celetron N5150 with 16G of RAM) and I can easily run Llama Mistral or Phi3 on it) although it is a bit slow 🙂 For most of my learnings I do use a 4-year-old Dell Precision 5540 with 32G RAM and the onboard NVIDIA Quadro GPU

How can I run LLM locally on my machine?

Here’s a list of applications that allow you to run large language models (LLMs) locally on a Windows (Mac or Linux) device:

PLEASE NOTE that this is just a sampling of what I have used — There are many others, some specialized.
Update As of 10/20/2024: You can now “compile” an LLM and create a chatbot / listening stand along application! See: llamafile( by Mozilla AI) lets you distribute and run LLMs with a single file. (announcement blog post) exciting!

Some of these applications are available as Docker containers. Docker Desktop for Windows (Mac, ARM or Linux) is one of the best ways to try out new applications without affecting or modifying your base OS!

Application	Description
Ollama	Command-line (CLI)tool for running LLMs with ease and flexibility. (Installation instructions)
AnythingLLM	All-in-one AI application that can do RAG, AI Agents, and much more with no code or infrastructure headaches. ( Docs here)
LM Studio	Integrated environment for experimenting with LLMs locally. (Docs here)
Open WebUI	Web-based interface for running various LLMs locally. It supports various LLM runners, including Ollama and OpenAI-compatible APIs. (Docs here)
H2O LM Studio	H2O LLM Studio, a framework and no-code GUI designed for fine-tuning state-of-the-art large language models (LLMs). (Ubuntu 16.04 with recent Nvidia drivers.
GPT4ALL	Framework and chatbot application for all operating systems. We can run the LLMs locally and then use the API to integrate them with any application, such as an AI coding assistant on VSCode.
Jan.AI	Jan is an open source ChatGPT-alternative that runs 100% offline.