Most importantly, I want to learn. I like trying to stay one step ahead of the field and make this understandable for others. With machine learning (ML) and AI, it’s a bit easier than many other areas I’ve studied. This is because I can use AI tools to help me understand and troubleshoot my work.

I’ve started accessing pre-trained models or creating and running them on my local computer and network. Normally, when you use an AI assistant like Siri, Alexa, or ChatGPT, the heavy computing work to make them understand you and give responses happens on big, powerful computers in the cloud, which are owned by companies like Apple, Amazon, and Anthropic.

Running AI models “locally” means doing that heavy computing work right on your own personal computer, laptop, or even a small home server instead of those big cloud computers. Instead of going to a big restaurant to have your food cooked for you, you bring all the ingredients and cooking tools to your own kitchen at home to cook the meal yourself.

You get more control over the whole process when you do it locally on your own machine. But it also means your computer has to work much harder than if you just sent the order to the cloud and let those big computers do the heavy lifting.

So running AI locally lets you really learn how it works under the hood, customize things your way, and keep your data private. But your computer might get really hot and slow if it’s not powerful enough. NVidia, if you’re listening…give me a call. (＠＾◡＾)

As I learn more about machine learning and generative artificial intelligence (AI), I am most interested in open-source models and what they could mean in this space. I’m also thinking about some questions about cost. Data, privacy, security, dollars spent for services, power, and environmental concerns cost. In this post, I’ll document several compelling reasons to go the local route.

Learn By Doing

One of the biggest advantages of running models locally is getting your hands dirty with the code.

When you use a large language model (LLM) like ChatGPT, the AI is learning and improving by conversing with the user and trying to become more responsive to human interaction. That means that ChatGPT is learning from all of the questions and prompts.

Instead of just using a cloud API (application programming interface), you must download the model, set up the necessary libraries and dependencies, preprocess your data, and integrate the model into your application or workflow. This process forces you to understand what’s happening under the hood rather than treating the model as a black box. You’ll gain practical experience that can help demystify AI/ML concepts.

Model Customization

Most cloud services offer a limited set of pre-trained models or allow you to train custom models within certain constraints. When running locally, you have complete freedom to explore different model architectures, data preprocessing techniques, hyperparameters, and training regimens. Plus, I can learn what all of those things mean. 🙂

You can tune models specifically for your use case rather than being limited by a one-size-fits-all approach. For example, I’m learning how to customize a model that works best for my kids as they do their homework or a different model for students in my pre-service teacher education courses.

Data Privacy & Security

Even if encrypted, sending your data to a cloud service raises privacy and security concerns. When you run models locally, your sensitive data never leaves your computer or private network. This can be essential for applications dealing with trade secrets, personal information, or any other confidential data you cannot risk exposing.

In addition, as you use generative AI tools, you’re training them using your questions, responses, and information. One of the complaints folks had as ChatGPT took off a little over a year ago was about where the data came from that was used to teach the models. Well, in many instances, it was scraped from online sources (Wikipedia, REDDIT, etc). Our interactions and use of these new tools are training current and future agents and tools.

Cost Considerations

While cloud services are convenient, they also come with recurring costs that can really add up for large-scale deployments or when leaving models running 24/7. Running locally avoids those cloud fees in exchange for a more substantial upfront cost on hardware. Over time, the local approach may be more cost-effective.

The tradeoff is that local processing power is limited by your hardware, whereas cloud services can leverage massive distributed clusters. However, improvements and cost reductions in areas like GPUs (graphics processing units) and TPUs (tensor processing units) make local AI/ML increasingly viable.

Electricity and Environmental Impact

The energy consumption for intense computation like AI training is directly linked to cost and environmental impact. As new technologies inundate our lives, we have little understanding of these tools and services’ true cost and impact, as that is often shifted elsewhere and hidden. This connection is vital for anyone concerned about their electric bills and carbon emissions.

Cloud providers often highlight their commitments to renewable energy, which is commendable. However, when you choose to execute these computations locally using your own energy-efficient devices, you gain significantly greater transparency and control over the energy sources and the overall energy efficiency.

Running AI training locally allows you to select hardware that is optimized for energy efficiency, potentially reducing the power required for processing. Additionally, managing the energy source becomes feasible, such as integrating solar panels or other forms of green energy into your infrastructure. This not only diminishes the reliance on non-renewable energy sources but also can lead to long-term savings and a reduction in carbon footprint.

While cloud computing remains a convenient choice for many due to its scalability and ease of access, the benefits of local processing, particularly in terms of energy consumption and environmental impact, make it a compelling consideration for businesses and individuals alike who are committed to sustainability.

Should you do this?

Will running AI/ML models locally be the right choice for everyone? Probably not.

Cloud services offer unbeatable convenience and scalability. But for learners, tinkerers, privacy/security sticklers, and those looking to optimize costs, going the local route is an intriguing option worth considering. The hands-on experience could pay major dividends in understanding how to develop and deploy AI effectively.