Run Local LLMs with Ollama and Spring AI

In the previous parts, we connected Spring AI with cloud-based AI models. But there is one important question: What if you don't want to send your data to an external AI provider? What if you want to: Run an LLM on your own machine Develop AI applications without API costs Work without an internet connection Keep sensitive company data private Experiment with different open-source models Build AI features locally before moving them to production This is where Ollama becomes very useful. In this article, we will learn how to run a local LLM using Ollama and connect it with Spring AI . We will build a simple real-world AI Customer Support Assistant using Java, Spring Boot, Spring AI, and Ollama. What We Are Building Our application will look like this: User | | HTTP Request v +---------------------+ | Spring Boot API | +---------------------+ | v +-------------+ | Spring AI | | ChatClient | +-------------+ | v +--------+ | Ollama | +--------+ | v Local LLM (Llama/Qwen) | v AI Response | v User The important part is that the LLM is running locally . There is no need to send every prompt to OpenAI, Anthropic, or another cloud provider. 1. What Is Ollama? Ollama makes it easy to run open-source LLMs locally. Instead of calling a remote API like: Spring Boot | v OpenAI API | v Cloud LLM we can run: Spring Boot | v Spring AI | v Ollama | v Local LLM Ollama can run models such as: Llama Qwen Gemma Mistral DeepSeek and many other compatible models The exact models available change over time, so always check the Ollama model library before choosing one. 2. Why Run an LLM Locally? Imagine you are building an internal HR application. Employees may send questions such as: What is our maternity leave policy? or: What is the process for requesting annual leave? You may not want internal company information leaving your infrastructure. A local LLM can help: Employee | v Spring Boot | v RAG / Business Logic | v Ollama | v Local LLM This can provide a useful privacy boundary. However