Direct answer: choose a RAG company by its ability to prove an end-to-end knowledge reliability chain. It must identify authoritative sources, keep content current, preserve permissions, retrieve relevant evidence, evaluate responses and operate failures in production. A demo that answers five questions about one PDF does not prove this capability. Before hiring, require a real question set, expected answers, citations, access tests, metrics, logs and a specific update and support plan.
Retrieval-augmented generation, or RAG, connects a language model to external information. It can make answers more useful for policies, products, contracts, customer support and internal knowledge. But RAG does not automatically turn disorganized documents into truth. Quality depends on every link between the original source and the answer shown to a user. Provider selection should therefore begin with decision risk and knowledge governance, not the selected model or vector database.
The MAKINAI Knowledge Reliability Chain
Compare proposals through six links: authority, freshness, permission, retrieval, response and operations. The final score is not a simple average; the weakest link constrains the system. For every link, request reproducible evidence in your environment and run a break test. A mature provider shows how the system fails safely when a source disappears, a permission changes, a question is ambiguous or a document attempts to manipulate the model.
1. Authority: who decides which source counts?
Start with a source inventory and accountable owners. The provider should distinguish official records, copies, drafts, messages, external content and informal knowledge. Request rules for priority, effective date, jurisdiction, language, duplication and conflict. NIST recommends documenting how generative AI systems are adapted, including retrieval and data provenance. Without lineage, a confident answer can silently combine incompatible versions of a policy.
- Minimum evidence: source catalog; owner by domain; version and effective date; conflict rule; required metadata; process to approve or withdraw content. Break test: insert two contradictory policies and verify that the system selects the current authority or communicates uncertainty.
2. Freshness: when does the index stop matching reality?
Ask how documents are ingested, transformed, chunked, indexed and removed. The proposal should state frequency, latency, change detection, deletion handling and reconciliation between the source repository and index. An append-only update process creates knowledge ghosts: revoked passages remain retrievable. Require a freshness indicator by source and a procedure for urgent corrections.
The provider must also explain handling of tables, images, attachments, spreadsheets and scanned pages. A pipeline that performs well on clean text can lose conditions, headers or critical notes. Run the proof with representative business material rather than a sample prepared by the vendor.
3. Permission: does the answer respect who asked?
Access control must travel with the document, retrieved passage and answer. Ask whether permissions are applied before retrieval, how groups and identities are synchronized and how changes propagate. A filter applied only after search can expose titles, passages or the existence of restricted documents. Include cross-department, hierarchy, customer and country tests.
- Evidence: identity model; permission inheritance; tenant isolation; query logs; retention policy; negative access tests; user-offboarding procedure. Break test: remove access to a document and measure how quickly it stops influencing every possible response.
4. Retrieval: does the system find the right evidence?
Microsoft describes RAG as a pipeline where a query moves through search, context assembly and generation. Every stage needs evaluation. Require retrieval measures such as coverage of relevant documents and precision of returned passages, segmented by domain, language and query type. Compare lexical, vector and hybrid search when appropriate. Do not accept one aggregate success rate that hides weak business areas.
Test acronyms, similar names, long questions, dates, negation, tables, multilingual queries and unanswerable requests. If the design uses agentic RAG, evaluate tool selection, number of calls, latency and autonomy boundaries too. Added complexity is justified only when it improves a measured task.
5. Response: is the conclusion supported, and does it know when to stop?
Evaluate correctness, relevance, completeness, context utilization and groundedness, dimensions identified in Microsoft evaluation guidance. Every material claim should point to a citation the authorized user can open. Define when the system answers, asks for clarification, presents alternatives or refuses. In legal, financial, clinical or operational settings, a calibrated refusal can be safer and more valuable than a high answer rate.
Build an evaluation dataset from real questions, expected answers, accepted sources, impossible questions and adversarial cases. Hold out a set the provider does not use during development. Record the model, prompt, index and configuration for every run because outputs are nondeterministic and small changes can shift results.
6. Operations: who detects degradation after launch?
Google Cloud’s reference architecture shows that a production RAG application spans ingestion, storage, retrieval, model and infrastructure. In practice, someone must monitor every part: update delays, connector failures, quality drops, cost, latency, unauthorized access and model behavior changes. Require dashboards, alerts, named owners, severity levels, SLAs, rollback and a recurring evaluation cadence.
Security must extend to retrieved documents. OWASP describes indirect prompt injection where malicious repository content influences model output. Providers should treat sources as untrusted data, separate instructions from content, constrain tools, filter inputs and outputs and test adversarial documents. RAG can reduce some factual errors while creating its own attack surface.
How to score providers: six ratings and a break test
Give each link zero to four points: zero means absent; one, a promise; two, a documented process; three, pilot evidence; four, reproducible production capability. Require at least three in Authority, Permission, Response and Operations. Then run one break test for every link. Do not offset weak permissions with a polished interface or weak evaluation with sophisticated architecture. The winning proposal should reduce verifiable risk, not merely accelerate the first demonstration.
How to structure a decision-quality pilot
Choose one domain with identifiable sources, available owners and observable value. Prepare fifty to one hundred representative questions, including unanswerable and access-restricted cases. Set acceptance criteria before testing: retrieval quality, source support, refusal, latency, cost and update time. Run the pilot with real users, document failures and estimate ongoing operating effort. The pilot ends with a decision about scope, controls, economics and ownership—not with a demo video.
- Minimum outputs: knowledge inventory; architecture; identity map; update pipeline; evaluation dataset; segmented results; risk register; support model; unit economics; documentation and transfer plan.
Red flags and next step
- The provider starts with the model instead of sources; promises to eliminate hallucinations; skips unanswerable questions; leaves permissions for later; citations cannot open the source; there is no evaluation dataset; revoked content remains indexed; security covers login only; cost estimates omit volume; production depends on undisclosed manual work.
Use this chain to compare bidders and combine it with the general scorecard at https://makinai.co/insights/en/how-to-choose-ai-implementation-company-brazil-scorecard. Formalize requirements with https://makinai.co/insights/en/how-to-write-rfp-ai-services. When the challenge is turning data and content into an operable intelligence system, visit https://makinai.co/services/en/data-content-intelligence-systems.