Their Lawyers Banned the Cloud. The On-Prem LLM Cost 81% Less.
A large insurance group in the DACH region had a working AI pilot its own lawyers would never let into production. The answer was an on-prem LLM: Qwen3-32B on two GPUs in the client's own building, with no outbound internet access. Three-year cost fell 81%, the hardware paid for itself in 3.9 months, and blind expert review put answer quality within 1.2 points of the frontier model.
Key Takeaways
- The blocker was never cost. No external AI provider could ever be approved for this workload
- Qwen3-32B in AWQ 4-bit on two NVIDIA L40S GPUs, served with vLLM behind an OpenAI-compatible internal endpoint
- 81% lower three-year cost: 75,580 euro in-house against a projected 392,400 euro on an external API
- 93.4% accepted answers against 94.6% for the frontier model, and better citation faithfulness at 97.1% against 94.8%
- Six weeks to production, with DPIA, DORA assessment, penetration test and legal sign-off done before go-live
The client is a large insurance group headquartered in Germany, writing business across the DACH region and supervised by BaFin, Germany's financial supervisory authority. The company is not named, and no policy, claim or customer data appears here.
The Challenge: A Pilot That Worked and Could Never Ship
The group had already run a proof of concept against an external frontier API. It worked, and claims and underwriting staff saved real time. The problem was how they had to use it: employees stripped personal data out by hand and pasted in only fragments, because nothing else was permitted.
That is a fine way to run an experiment and a useless way to run an insurance operation. The real workflow needs complete claim packs, full policy documents, underwriting manuals and the personal data inside all of them. When the team asked to go to production with real documents, legal and information security said no, for four reasons:
- GDPR obligations covering policyholder and claimant personal data
- BaFin expectations and DORA requirements for ICT risk management
- Non-disclosure agreements with corporate clients, forbidding any transfer of their documents to third parties
- Internal data protection rules, including an outright ban on sending documents or prompts to external AI providers
The condition that came back decided the architecture: production AI was possible only if the data physically never left the client's own infrastructure.
The Approach: Start From What Legal Can Approve
Most AI programmes design the system first and hand it to compliance at the end. That ordering killed the pilot. Here the first architecture workshop included information security and data protection, and the constraints were fixed before any model was chosen: no data egress, no unrestricted outbound internet, role-based access enforced at retrieval, a full audit trail for every answer. An on-prem LLM is a harder engineering problem than an API call, but it was the only version of the problem with a signature at the end.
The Solution: Four Layers, All Inside the Building
Serving: Qwen3-32B on two GPUs
The production model is Qwen3-32B Instruct, quantized to AWQ 4-bit and served with vLLM behind an OpenAI-compatible internal endpoint, tensor parallel across two GPUs. That endpoint matters more than it sounds: internal applications talk to it exactly as they would to a commercial API, so the rest of the estate needed no bespoke integration work.
The hardware is a dedicated inference node with two NVIDIA L40S 48 GB cards, 512 GB of RAM, mirrored NVMe storage and redundant power, in the client's own server room in Germany, with no direct outbound internet access. Weights, containers and updates arrive through a controlled deployment process.
Retrieval over the documents that actually matter
The model is reached through an internal RAG platform, not a raw chat box, indexing underwriting manuals, claims guidelines, policy and product documentation, the knowledge base and selected SharePoint repositories.
Every answer cites the specific internal documents and pages it drew on. In insurance that is not a nicety. A claims handler who cannot show where a rule came from cannot use the answer, and an auditor who cannot trace it will not accept it.
Access control and auditability
Retrieval is bound to corporate SSO and role-based access, so the model cannot surface a document the user is not entitled to see. That is enforced in retrieval, not in a prompt. For every request the platform stores user identity, retrieval sources, model version, prompt template version, response and timestamps, which is enough to reconstruct afterwards why a particular answer came back and what it rested on.
Security and controlled deployment
Encryption at rest, TLS internally, secrets in the group's existing secrets manager, network segmentation, automated vulnerability scanning, and a controlled path for putting a new model version into service. None of it is exotic. It is the part AI pilots skip and then cannot retrofit.
The Results: 81% Cheaper, and Actually Approved
The system serves around 42,000 inference requests a day at a peak concurrency of 34. An interactive copilot for claims and underwriting staff shares the endpoint with batch work: summarizing claim packs, document classification, fact extraction and first-draft case summaries. Batch jobs run through a separate queue, so bulk processing never slows down someone waiting on an answer.
The cost comparison was calculated from actual request and token volume after launch, not a forecast:
- A comparable external frontier API at this volume: about 10,900 euro a month
- On-prem capex: 38,500 euro, covering the GPU server, both L40S cards, storage, networking and three years of hardware support
- On-prem running cost: about 1,030 euro a month, including power and cooling, monitoring, backup and hardware support
- Over three years, 75,580 euro in-house against 392,400 euro on the API: a saving of 316,820 euro, roughly 81%
- Hardware payback at current volumes: 3.9 months
Cost was not why this project happened, and it is worth saying so, because the economics are the easiest part to quote and the least important part of the decision. The external API was never going to be approved at any price. The 81% is what the client found waiting once they had solved the problem they actually had.
Quality: Measured, Not Assumed
Before rollout the team built an evaluation set of 600 real tasks with personal data removed: policy and underwriting questions, claims scenarios, summarization, information extraction, and questions needing several documents at once. The client's own domain experts scored the answers blind.
- External frontier model: 94.6% accepted answers
- On-prem Qwen3-32B: 93.4% accepted answers, a gap of 1.2 percentage points
- Citation faithfulness, external model: 94.8%
- Citation faithfulness, on-prem system: 97.1%
The citation result is not a claim about model quality. A 32B open-weight model is not stronger than a frontier model. The retrieval pipeline and reranking around it were tuned for these specific documents, and on the measure that decides whether an insurer can use an answer at all, that tuning mattered more than raw model capability.
Single-stream generation runs at about 72 tokens per second, and about 54 per stream under load. The p95 time to first token is 1.3 seconds, p95 end to end is 7.1 seconds, so staff see an answer start within about a second.
The pilot proved the technology worked. Legal proved it could never ship. Those turned out to be two different problems.
Compliance: What It Took to Get Signed Off
Four blocks had to close before production: data residency, with prompts, documents, embeddings and outputs staying inside the client's infrastructure on servers without unrestricted internet egress; access control through SSO and role-based collections; auditability through the per-request record above; and security.
Completed before go-live: a DPIA, an internal information-security review, a DORA ICT risk assessment, a penetration test, a legal review and an architecture approval. The final penetration test closed with zero critical and zero high-severity findings after remediation, and the deployment carries written sign-off from the legal, information security and data protection teams.
How It Was Built in Six Weeks
Six weeks from architecture workshop to production, about 980 hours of engineering effort, and a team of four: an ML and AI engineer, a backend engineer, a DevOps and security engineer and a tech lead, with the client's QA and compliance specialists part-time.
Claude Code was part of how the team worked, following the agentic development process we use on every engagement: vLLM integration, RAG service boilerplate, deployment scripts and infrastructure as code, the evaluation harness, regression tests, observability dashboards and refactoring the retrieval pipeline. The evaluation framework and the infrastructure code are where it saved most. The team's own estimate is that agentic development cut engineering effort by 25% to 30%, turning an eight-week build into six.
One rule was absolute, because it is the obvious question: Claude Code helped build the system and never saw the data the system exists to protect. It ran on development machines in an isolated environment against synthetic and anonymized test data only.
Weighing the same decision? The companion article works through the numbers: what a self-hosted LLM actually costs, alongside a walkthrough of setting up a private LLM on premise.
Frequently Asked Questions
Can an on-prem LLM really match a frontier model?
Close enough to matter. On a 600-task evaluation set scored blind by the client's own underwriting and claims experts, the in-house Qwen3-32B deployment reached 93.4% accepted answers against 94.6% for the external frontier model. On citation faithfulness, which is what an auditor asks about, the in-house system was better: 97.1% against 94.8%.
What does an on-prem LLM cost compared with an API?
For this workload, 38,500 euro of hardware plus about 1,030 euro a month, against roughly 10,900 euro a month for a comparable external API at the same measured volume. The hardware pays for itself in 3.9 months and saves about 81% over three years.
Does self-hosting mean the AI vendor never sees the data?
That is the whole point. The inference node has no unrestricted outbound internet access, and prompts, retrieved documents, embeddings and model outputs stay inside the client's own infrastructure.
How long does it take to move an AI pilot on-premise?
This one took six weeks from the first architecture workshop to production, with four Valletta engineers plus the client's QA and compliance specialists part-time. Running the compliance work in parallel with the build, rather than after it, is what kept it to six weeks.
Start With a Five-Day Discovery Audit
AI programmes in regulated industries fail in a predictable place: the pilot proves the technology, then cannot be approved, because the constraints were discovered after the architecture was fixed. The cheapest time to learn what your legal and security teams will sign is before anything is built.
In five working days, for a fixed fee of €2,000, two of our engineers map your workflow, measure where the hours and the risks sit, and hand you a costed, prioritized plan, whether or not you build it with us.
Book your five-day Discovery audit: vallettasoftware.com/discovery-audit