AI resource library: upload the documents, ask in plain language, get cited answers

Ingestion: from a file to searchable chunks

Uploads are parsed by format, PDF, Word, spreadsheets, CSV, plain text and HTML, or a URL is fetched and stripped to text. URL discovery is restricted to an administrator-managed domain allow-list, so a crawl cannot wander off your own material.

Text is split into overlapping chunks of about 1,200 characters with 150 characters of overlap, preferring paragraph or sentence boundaries when the cut point is already past halfway. Overlap is what stops an answer being lost across a page break, the classic reason naive document search misses the one paragraph that mattered.

Each resource tracks its own status through pending, parsing, ready or failed with an error message, along with its chunk count, tags and scopes. Files are stored in a dedicated bucket and served through signed URLs.

Hybrid retrieval, because meaning alone is not enough

Chunks are embedded at 1,536 dimensions and indexed for cosine similarity. Retrieval then runs two searches: vector similarity for meaning, and Postgres full-text ranking for literal wording. The two ranked lists are fused with reciprocal rank fusion rather than one being used as a tiebreaker.

This is the difference between "what is our policy on damaged returns" and "clause 7.3". A pure vector search handles the first and fumbles the second. Hybrid handles both, which is what document Q&A in a real business actually requires.

• Vector candidates and full-text candidates retrieved in parallel
• Ranks fused rather than one signal overriding the other
• Candidate pools deliberately oversized before reranking
• Six chunks returned to the answering model by default

Reranking and scoped material

When the hybrid pass returns more candidates than needed, a reranking model scores each chunk from zero to ten for relevance to the question, and only the best survive. Retrieving twenty and keeping six produces materially better answers than retrieving six and hoping.

Resources also carry scopes, so a document can be made available to specific AI surfaces only, the internal assistant, support chat, product copy, category copy, brand copy or assessment work. Supplier contracts can inform staff answers without ever being reachable by a public-facing assistant.

Citations and a retrieval log you can audit

Retrieved material is formatted into a numbered reference block with each source's title and URL, and the answering model is instructed to cite those sources. Citation is a prompt-level instruction over verified retrieved text, the text is real and traceable, which is the part that matters when someone challenges an answer.

Every retrieval is logged: the user, the calling feature, the scope, the query, the mode, whether reranking ran, the candidate set, the final chunk identifiers, latency and any error. That log is browsable in the admin, which means when someone says the AI gave a bad answer, you can see exactly what it was given.

Why this beats a shared drive and a search box

Because the same infrastructure serves several surfaces. One ingested document can inform staff answers, support chat and content generation, scoped separately, with one audit trail and one place to remove a document when it is superseded.

For franchise networks this is usually the highest-value module we deploy: consistent answers across every branch, with visibility into what people are actually asking.