Blog / Enterprise AI / LLMs & Agents
ENTERPRISE AI · LLM & AGENTS
Local LLMs in Practice: How We Use Them at theBlue.ai
Author: Aleksandra Osztynowicz / Published: August 2026
A default approach has emerged around large language models (LLMs): whenever there is a problem to solve with AI, we send data to ChatGPT or a similar cloud tool. In many cases, this choice is justified, but not always. For a whole class of tasks, running LLMs locally, an open-source model running on infrastructure you control, turns out to be cheaper, safer, and easier to control than a service accessed through an external API.
This is not a theoretical claim. It is based on real systems that we have built and that we use in our daily work. Below we describe what it looks like in practice, where the local models really provide an advantage, and where they hit their limits.
What Is a Local LLM?
A local LLM is an open-source model, for example from the Gemma, Llama or Qwen families, running on our own hardware rather than on an external provider server. For a business, two consequences matter. First, data never leaves the company’s infrastructure. Second, there is no per-query fee. Together, these two factors change both the cost profile and the risk profile of a wide range of applications.
Three Systems, Three Lessons
In practice, we run several such systems. Below we describe three of them, starting with the one that best illustrates what it means to fit a model to a task, our tender portal.
1. Tender Portal: Screening Public Procurement at Scale
Our first example is an internal portal that automatically searches for public tenders in our areas of interest and downloads everything available about them: descriptions, requirements, attachments, and supporting documentation. Once the data has been filtered, the local model Gemma 4 26B (26 billion parameters) with an MTP (multi-token prediction) layer takes over. It handles two stages of the process:
- Eligibility checks: The model reads the header of each candidate tender, title, buyer, category, attachment names, and renders a short verdict: whether the tender fits our company profile, with a confidence level and a brief explanation. After this stage, only relevant tenders remain on the list.
- Documentation analysis: Once the documentation and attachments have been downloaded, the local LLM analyzes them and returns a structured summary: a short description of contract, financial, staffing, and experience requirements, required certificates and attestations, the amount and form of the bid bond, key dates, risks (such as contractual penalties), and a final recommendation, RECOMMEND / CONSIDER / REJECT, with justification.
The scale here is real. In a single day, the LLM processes between 15 and 20 million tokens, the equivalent of roughly 15,000 A4 pages. Over a year, that adds up to 5.5 to 7.3 billion tokens. For comparison, running the same workload through GPT-5 would cost $6,875 to $9,125 for input tokens alone (based on OpenAI’s official pricing as of July 2026).
The choice of model was not accidental. In the portal, a user is actively waiting for the result; response time matters. On our hardware, Gemma 4 26B generates around 88 tokens per second. The larger 31B variant drops to 40. The 31B version is slightly more accurate, but once user experience is factored in, the 26B model is the right compromise for this application. It owes its speed to the MTP layer and a mixture-of-experts (MoE) architecture, both described in more detail below, which together accelerate token generation. It also offers a large context window, which lets us analyze an entire tender together with its attachments in a single pass, and it handles Polish well.
Match the model to the application, weigh accuracy, latency, context window, language support, and hardware footprint against each other, and the result, in this case, is that we can process large volumes of tenders without worrying about API costs or query limits.
2. Policy Insider: Where Per-Query Pricing Breaks Down
The tender portal shows the cost side of the argument. Policy Insider takes it to a different scale. Policy Insider is an AI-powered policy and regulatory intelligence platform. It helps public affairs, government relations, and strategic advisory teams monitor policy and regulatory developments at scale, so they don’t have to rely on manual tracking, spreadsheets, or fragmented tools. Here we use the local LLM Gemma 4 31B, to analyze content from a large number of policy-related websites, about 28 million tokens per day. For each site, the model preprocesses the content, generates a summary, and flags whether it matches a given topic. The sites classified as relevant then feed further outputs: reports, newsletters, and heatmaps showing where change is most intense.
We deliberately use the larger, more accurate model with 31 billion parameters here rather than the faster 26B variant from the previous example. Policy Insider processes data in cyclical background jobs, so it does not matter whether a batch takes one minute or thirty, what matters is classification accuracy, since the quality of the resulting reports depends on it. The principle is the same as before: we choose the model to fit the task, not the other way around.
This is also a case where the economics of a paid API stop adding up. Millions of tokens per cycle would translate into high, constantly growing costs and ongoing wrestling with query limits. With a local LLM, the cost of processing another website is effectively zero: we pay for the hardware once, and scale is no longer a financial constraint.
3. Internal Chatbot With RAG: Data That Never Leaves the Company’s Infrastructure
Cost and scale are one part of the story. Data control is the other. The third system is built for our own needs: a chatbot that searches our internal documentation and answers the team’s questions. Instead of manually digging through dozens of documents, we ask a question in natural language, and the model returns an answer along with a pointer to the source it used. The design follows the RAG (retrieval-augmented generation) pattern: the system first retrieves fragments related to the question from our documents rather than relying on knowledge acquired during training; the model’s job is to understand those retrieved fragments and compose an answer from them.
We run the system locally, on our own GPUs, this time with a model from outside the Gemma family, Qwen3 30B. This is the clearest illustration of a principle that runs through all three implementations: match the model to the task, and there is no need to stay within a single model’s family. Qwen3 handles conversation well, copes with long documents, and despite its nominal 30-billion-parameter size it stays fast because each query activates only a fraction of its parameters (mixture-of-experts architecture). As a result, it remains responsive even when many people query it at once, and it can draw on large sections of our documentation for every answer.
But what matters most is where the work happens: our internal knowledge, projects, proposals, know-how, never leaves the company for an external provider. For us, this is a matter of peace of mind; for clients in regulated sectors such as finance, insurance, government or healthcare, it can be a non-negotiable condition before AI adoption is even on the table.
Limits and Trade-offs
Local LLMs are not a universal solution. It’s important to be aware of their limits:
- Upfront cost: Hardware is a capital investment. It pays off at sufficient scale and over a long enough time horizon; for small scale or occasional use, a cloud API can be the cheaper option.
- Hardware as a ceiling: Because the model runs on our own machines, we are limited by their compute, memory and architecture. The largest and most capable models may simply be impossible to run on a given setup. In the cloud, none of that matters, we can call an external provider's API from an old laptop with nothing more than an internet connection, because all of the compute lives on the provider's side.
- Quality gap on the hardest tasks: For the most complex problems that require deep reasoning, leading commercial models can still deliver better results. The trick is matching the model to the task, because not every problem needs the highest tier of reasoning.
- Ongoing maintenance: The model has to be hosted, monitored, and updated. That is a real operational load, not a one-time deployment.
How to Run LLMs Locally: What to Look For
The effectiveness of a local deployment depends as much on how the model is served as on which model you choose. A handful of technical decisions in a local LLM setup have a disproportionately large impact on cost, quality and speed.
- Quantization: The model's weights can be stored in lower precision, for example 8-bit or 4-bit instead of the default. Lower precision means lower GPU memory usage and, usually, faster inference. The more aggressive the quantization, the bigger the savings, but also the more pronounced the potential drop in output quality. The key is to find the level at which the model runs faster and takes up less memory while the quality loss stays acceptable for the task at hand. That is why we always validate the chosen compression level with quality tests on our own data.
- MTP (multi-token prediction) layer: A standard model generates an answer one token at a time, which caps its speed. An MTP layer lets the model predict several tokens in a single step, speeding up generation without hurting quality. The size of the effect depends on the model: in our test with the model used in the tender portal, adding the MTP layer raised throughput from around 72 to 88 tokens per second; with the larger 31B variant, it went from 16 to 40, more than doubling. The trade-off is that an MTP layer is not available for every model and slightly increases its size, so it is not a default solution for every situation.
- Mixture-of-experts (MoE): Instead of running every parameter for every token (dense architecture), an MoE model is split into many specialized sub-networks, "experts", and a small router picks only a few of them for each token. The model may have tens of billions of parameters on paper, but any single query activates just a fraction of them. In practice, this means an MoE model of a given nominal size runs much faster and cheaper than a dense model of the same size, while retaining most of the quality benefits of being large. The model used in our tender portal is a good example: despite its 26 billion parameters, it activates only about 4 billion for any single query, which is what makes it more responsive. The trade-off sits on the hardware side: all the experts still have to fit in GPU memory even though only some of them fire at a time.
- Context window and model cache: The longer the text handed to the model, for example, an entire tender with attachments, the more GPU memory is taken up by the cache where the model stores the context it has processed so far (typically called the KV cache). The context length actually available in practice depends not only on the model itself but also on how much memory is left after loading its weights. This is one of the factors that has to be planned from the start.
- Serving software: The serving stack determines how much throughput the same hardware delivers. Some solutions (vLLM, SGLang) are optimized for high throughput and can handle many parallel queries at once, which fits high-scale production use. Others (Ollama, llama.cpp) are easier to spin up and work well for prototyping or smaller deployments, but they trade off performance under load.
- Handling many queries at once: When the model is used by many people or processes simultaneously, the key capability is grouping parallel queries into efficient processing batches, a mechanism known as continuous batching. Mature serving software handles this automatically, and it is the main reason the same hardware can serve several times more queries than sequential processing would allow.
- Hardware selection: The main constraint is usually the amount of GPU memory. It has to accommodate the model's weights, the context window mentioned above, and headroom for parallel queries. If we have a choice of hardware (we are not tied to what is already on hand), the right starting point is calculating how much memory is needed to run the desired model at the intended quantization level with the expected load, so that memory size does not become the bottleneck.
These decisions do not have to be made in isolation, quantization, serving stack, and hardware sizing all interact, and getting one wrong tends to show up as poor performance somewhere else in the chain.
When the Local Model Makes Sense, and When Not
The three systems described earlier each hit a sweet spot where running a model locally is genuinely the better option. But there are just as many cases where a cloud API is the right answer.
A local model tends to make sense when:
- Data cannot leave the company: Personal data under GDPR, medical records, financial information, trade secrets, or documentation subject to industry regulations (finance, insurance, healthcare, public administration) either cannot be sent to an external provider at all or can be sent only under safeguards that make the whole operation heavier than running the model in-house. In regulated sectors, local deployment is often not an optimization, it is a precondition for the project to happen.
- Volume is high and predictable: At millions of tokens per day in a repeatable pattern, per-query API costs quickly outpace the amortized cost of your own hardware. The tender portal and Policy Insider both fall here.
- You need full control over the model: Fine-tuning on your own data, freezing the model version so behavior does not drift after a provider's update, running custom quantizations, or building non-standard serving setups, with a closed API, all of this is either impossible or heavily restricted.
A cloud API tends to make sense when:
- Volume is small or unpredictable: For occasional use, hardware amortization never catches up with pay-per-query pricing.
- You need the highest possible reasoning quality: For the hardest tasks, frontier commercial models still have the edge. If quality on the toughest queries decides the value of the whole system, that is where a cloud model earns its cost. That said, local models keep closing the gap, Kimi K3 is a good example, so this argument may carry less weight over time.
- You do not have people to maintain the infrastructure: A local deployment is not a one-time project, someone has to keep the GPUs healthy, update the serving stack, and handle incidents. Without that capability in-house, an API is the most honest choice.
A Note on Data Security
The security argument for local models is stronger than “your data does not leave the building”, and also more nuanced than that phrase suggests.
A local deployment keeps data inside infrastructure you control, under your access policies, your logging, your retention rules. There are no third-party terms of service to reconcile with your own compliance obligations, no ambiguity about whether queries might be used for future model training, and no dependency on the provider’s security posture. For enterprise deployments in regulated sectors, this often shortens the compliance conversation from months to days.
But a model running on your own hardware is not secure by default, it is only as secure as the environment around it. Access has to be authenticated and audited, prompts and outputs may still contain sensitive data that needs to be handled with care, and RAG systems inherit the access controls of the underlying document store, a chatbot that indexes files a user should not see will happily surface their contents.
The right way to think about it: a local model removes a large category of risk (external data egress) and replaces it with a smaller, more familiar one (application and infrastructure security). For most organizations, that is a favorable trade, but only if the second category is taken seriously.
Key Takeaways
- →Local LLMs earn their place on volume, sensitivity, or both. At millions of tokens per day, per-query API costs outpace amortized hardware. Under sector regulation (finance, insurance, healthcare, public administration) or GDPR, a local model is sometimes the only option that clears legal review at all. Without those requirements, a cloud API is usually the honest choice.
- →Match the model to the task, not the other way around. We run Gemma 4 26B where response time matters (users waiting, 88 tokens per second) and the larger 31B variant where accuracy trumps latency (background batch jobs). Model family should be a variable in the design, not a commitment made up front.
- →How you serve the model matters as much as which model you pick. Quantization, an MTP layer, mixture-of-experts, and the serving stack can more than double throughput on the same GPUs. Adding an MTP layer alone raised our Gemma 4 31B from 16 to 40 tokens per second without touching quality.
- →Local removes one class of risk and adds another. Data no longer leaves your infrastructure, which typically shortens security reviews from months to days. In exchange, you own application security, access control, and the fact that an LLM inherits the permissions of the document store it queries. It's a favorable trade only if you take the second category seriously.
- →A local deployment is an operational commitment, not a one-off project. Someone has to keep the GPUs healthy, update the serving stack, and respond to incidents. Without that capability in-house, a managed API is usually the more honest starting point, even at higher unit cost.
How We Can Help
At theBlue.ai, we deploy local LLMs for enterprise use, both for our clients and inside our own company. We’ve walked this path on our own systems and delivered it for others, so we know where the real trade-offs sit and when a local model is genuinely the right answer versus a cloud API or a hybrid setup.
If you’re weighing this for a specific workload, our AI Discovery Workshop (1 to 3 weeks) is designed to answer exactly that question and hand you an architecture proposal you can act on. Get in touch →
About the Author
Aleksandra Osztynowicz, AI Engineer, theBlue.ai
Aleksandra has been building custom AI solutions at theBlue.ai since 2021, with a focus on agentic implementations and local LLM deployments that precisely fit enterprise needs. As an AI Engineer, she helps organizations automate their processes with production-grade systems, from on-premise open-source models to RAG-based internal knowledge bases for regulated industries, and continuously expands her knowledge in the rapidly changing world of artificial intelligence.In her articles, she shares practical experience from real enterprise AI projects and shows that deploying AI in companies doesn’t need to be complicated.