Blog · 24 September 2026 · 7 min read
How to connect a local AI to your company documents (RAG)
The real value of AI in a company comes when it answers with the company's own information: procedures, price lists, contracts, manuals. The technique is called RAG. Here's how it works and how to set it up properly on a local AI.
In short: RAG (retrieval-augmented generation) splits documents into small chunks, indexes them with an embedding model and, for each question, passes only the relevant chunks to the AI. In Open WebUI you do this by creating a Knowledge base and linking it to a model. For Italian, it pays to replace the default embedding model with a multilingual one, such as bge-m3 or qwen3-embedding, and to increase the model's context.
How it works, without the jargon
- Preparation: each document is split into chunks of text (in Open WebUI, by default, around 1,000 characters with 100 of overlap).
- Indexing: an embedding model turns each chunk into a numerical "fingerprint" that represents its meaning.
- Search: when you ask a question, the system looks for the chunks closest in meaning (the top 3 by default).
- Answer: the model receives the question together with those chunks and answers, citing its sources.
Everything happens on the company computer: neither the documents nor the questions leave the company.
Setting it up in Open WebUI
- Download a multilingual embedding model:
ollama pull bge-m3(1.2 GB) orollama pull qwen3-embedding:4b(2.5 GB). - In Admin Panel → Settings → Documents choose Ollama as the embedding engine and select the model you downloaded. Open WebUI's default model is designed mainly for English.
- In Workspace → Knowledge create a knowledge base for each topic (for example "Quality procedures", "Price lists 2026") and upload the documents.
- Link the Knowledge base to a model in Workspace → Models, or call it up in chat by typing
#. - Increase the model context (
num_ctx): with the default of 4,096 tokens on GPUs under 24 GB, the retrieved chunks may not fit.
Important: if you change the embedding model after uploading your documents, they have to be re-indexed. Choose it before uploading your archive.
Why the answers are sometimes wrong
- Scanned PDFs: if the PDF is an image with no text, the AI reads nothing. Text recognition (OCR) is needed first.
- Tables and price lists: once split into chunks they lose their structure. For tabular data, a direct connection to the management system or database is often better than RAG.
- Duplicate or outdated documents: if the archive holds three versions of the same procedure, the AI may quote the wrong one. Upload only the valid versions.
- Too many documents in a single base: smaller, topic-based knowledge bases give more precise answers.
- Questions that are too vague: "tell me everything about customer X" works worse than "what are the payment terms in the contract with customer X?".
Permissions: who can see what
An AI connected to documents sees everything you give it. Before uploading contracts, staff data or confidential information:
- create separate knowledge bases per department or confidentiality level;
- use Open WebUI groups to decide who can use which knowledge base;
- don't upload personal data you don't need: the GDPR data minimisation principle applies to AI too.
When it's worth getting help
Want it all in a PDF to print or share with colleagues? Download the complete guide for free: hardware, models, installation, security and a 20-point checklist.
A trial on one computer takes an afternoon. Taking it into production for a company is a different job: accounts and permissions per department, secure remote access, backups, updates, connection to management systems and staff training (which also helps with the AI Act).
If the trial has convinced you and you want a reliable installation, get in touch: we'll start from what you've already tested. Dabryx consultancy starts at €100 per hour.
Frequently asked questions
What is RAG?
It's a technique that lets an AI model answer using documents it didn't know: the system looks for the relevant passages in the archive and passes them to the model together with the question.
Does the model need retraining on company documents?
No, in most cases. RAG is simpler, cheaper and updates immediately when you change your documents; training (fine-tuning) is for specific style or format requirements.
How many documents can be connected?
Even thousands. The practical limit is quality: topic-based knowledge bases, up-to-date documents and readable text matter more than quantity.
Want an estimate for your company?
Tell us how many people will use the AI and what for: we'll tell you which configuration you need.
Response times
Within 2 hours on business days (9:00–18:00) for clients with a support contract.
AI consulting
From €100 per hour, on your company's servers or Macs.
Where we work
All over Italy, remotely and on site.