Solution / Enterprise AI search
Useful answers. Traceable sources.
Enterprise AI search, often built with retrieval-augmented generation (RAG), lets people query organizational knowledge in natural language and get answers grounded in real documents. Quorlytic enforces access permissions before retrieval reaches a model, ties every answer to a traceable source, and evaluates retrieval and answer quality separately rather than trusting a fluent response.
By QuorlyticUpdated
Help teams find and use organizational knowledge while preserving access boundaries. Build retrieval, citations and evaluation into the architecture from the start.
What changed in our returns policy?
Grounded answer
The returns window moved from 14 to 30 days for standard orders, and the restocking fee for opened electronics was removed. The change applies to orders placed after 1 March.
Why this answer
Retrieved from the current policy version and the operations memo that announced the change. A superseded handbook entry was excluded because a newer version replaces it.
Sources
- 01Policy v4current
- 02Operations memo1 Mar
- 03Support handbookupdated
Problem, approach and outcome.
| Problem | Choose a collection and user group with clear permissions and ownership. Define what the system should answer, what requires a source and when it should decline. A small evaluation set should include common, ambiguous and unanswerable questions. |
|---|---|
| Approach | Connect documents to their version, owner, access policy and canonical location. Apply authorization during retrieval before content is sent to a model. Re-index updates and propagate deletion so stale or withdrawn material does not remain available. |
| Typical outcome | Knowledge-source and permission inventory |
The approach
- 01
Start with a bounded knowledge task
Choose a collection and user group with clear permissions and ownership. Define what the system should answer, what requires a source and when it should decline. A small evaluation set should include common, ambiguous and unanswerable questions.
- 02
Preserve source identity and access
Connect documents to their version, owner, access policy and canonical location. Apply authorization during retrieval before content is sent to a model. Re-index updates and propagate deletion so stale or withdrawn material does not remain available.
- 03
Evaluate retrieval and answers separately
Check whether the right permitted source was retrieved, then whether the answer is supported by it, and keep the two scores apart: a correct document can still be summarised incorrectly, and a fluent answer can hide a retrieval failure. Measure citation correctness against the version actually used, refusal behaviour, latency and operating cost, and include ambiguous and unanswerable questions in the evaluation set from the start. Re-run evaluations after changes to parsing, retrieval, prompts or models.
- Retrieval recall scored separately from answer quality
- Citations checked against the version actually used
- Unanswerable questions retained, not filtered out
- 04
Protect the workflow around the model
Treat retrieved text as untrusted input. Restrict tool access, log only what is necessary and preserve a human escalation path. A read-only search application should not silently become an autonomous system that changes business records.
- 05
Refusal behaviour is a feature
The system needs a defined way to decline: no permitted source found, sources conflict, or the question falls outside the corpus. A refusal that names the limitation is more useful than a confident answer without evidence. Define who reviews refusals and how a genuine gap becomes an ingestion or permission fix.
- 06
Freshness, deletion and revocation travel together
An index is a derived store, so it inherits the obligations of its sources. When a document is updated, withdrawn or its access changes, the index has to follow within an agreed window. Test the deletion and revocation paths deliberately rather than assuming re-indexing is sufficient.
- Update, withdrawal and permission-change paths tested
- Revocation window agreed with the source owner
- Indexed content traceable to a canonical source and version
- 07
Latency and cost are design choices
Retrieval depth, re-ranking, model choice and answer length each move latency and cost together. Set an acceptable envelope per task — an assistant answering an operational question has different needs from a background research tool — and measure cost per answered question rather than per token, so the trade-off stays visible.
- Latency and cost envelope agreed per task
- Cost measured per answered question
- Degradation path defined when the budget is exceeded
- 08
Operate the index like a data product
Monitor retrieval quality, unanswered rates, citation failures, freshness and cost after launch, with an owner for each. Version parsing, chunking, retrieval configuration, prompts and models together, and re-run the evaluation set before promotion so behaviour changes are visible before users find them.
What the work produces
- Knowledge-source and permission inventory
- Retrieval and citation architecture
- Task-based evaluation and adversarial tests
- Freshness, deletion and incident runbooks
Delivery stages
- 01Owned knowledge
- 02Index with permissions
- 03Retrieve & cite
- 04Evaluate & monitor
Test a permission change beyond the search result.
A correctly filtered search request can still sit behind an unsafe answer cache or a conversation that reuses old passages. In this hypothetical policy search, Reader A can access a restricted collection and Reader B cannot. The useful acceptance test repeats the same question across both readers, then changes A's access and follows every path that could reuse the answer.
Scope and assumptions: Use synthetic documents and test identities in an approved environment. Agree when a permission change must affect future requests and which identity information is authoritative. The record is a proposed application test, not a guarantee that a search product governs every cache, conversation or source link.
| Dimension | Request path | Expected behavior and evidence |
|---|---|---|
| First request | Ask the same question as A and B with empty application caches. Record the identity and entitlement decision used for each request. | A may retrieve the permitted document. B must not receive its passage, title or citation. Inspect the context sent to the model as well as the visible answer, using appropriately restricted evidence. |
| Warm answer cache | Let A populate a cached answer, then repeat the question as B. Matching question text does not establish equivalent access. | The application must evaluate whether B is entitled to the material before reusing it. Test the actual cache path; a correct direct-search result says nothing about a cache that bypasses that search. |
| Changed membership | Remove A's access using the approved test procedure, then make a new request through the existing signed-in session after the agreed propagation boundary. | Record the access-change event and the first request subject to the new decision. A stale entitlement or cached answer must not silently continue to authorize future disclosure. |
| Follow-up question | Continue a conversation that previously used the restricted passage. Distinguish information already displayed from content being sent into a new model request. | Apply the current access decision to reused context and generated follow-ups. Revocation cannot make a reader forget an earlier answer; it must not be represented as erasing a disclosure that already happened. |
| Citation access | Open the cited source as the reader making the current request, including after the access change. A citation is a pointer, not an entitlement. | Verify the source or document-serving endpoint enforces its own access rule. Record any temporary-link policy and exposure window; an answer being hidden does not make an independently readable link safe. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: B's direct retrieval returns no restricted documents, but a cached answer created for A includes the restricted policy title. The search filter passes its own check while the application fails the disclosure requirement. Keep the failed cache case in the acceptance set and repeat the identity, follow-up and citation checks after correcting that path.
- Microsoft's Azure search sample documents login and document access controls, with explicit configuration and identity requirements. Those controls are a useful reference for that sample; they do not certify the behavior of a separate application cache or chat history.
- Keep access-change timing, denied cases and recovery behavior with the operating owner. Use a deliberately permitted test collection and avoid copying restricted production passages into general debugging logs.
Access is only one reason a source may be unsuitable. Keep the source applicability and readiness record alongside these request-path checks.
Official references for this example:
Questions
Questions worth asking.
Does RAG prevent hallucinations?
No. Retrieval can provide relevant evidence, but the answer still needs evaluation and a safe failure mode. Test unsupported answers, incorrect citations and access-control boundaries explicitly.
How do we evaluate an enterprise search system?
Build an evaluation set from real questions, including ambiguous and unanswerable ones, then score retrieval and answering separately: was the right permitted source retrieved, is the answer supported, and does the citation resolve to the correct version. Re-run the set whenever parsing, retrieval, prompts or models change.
Your next data decision
Start with a clear problem.
Build what comes next.
Bring the business question.
We'll help shape the engineering brief.