Decision models in agentic RAG: nine checkpoints.

What does the decision layer add?

Separate writing from judgment. An assistant answering an expense-policy question may find documents, compare versions, draft a reply and propose a ticket. Each transition needs a rule. A fluent final answer cannot establish that those earlier choices were sound.

Jev is one implementation. TypeSafe’s decision API accepts supplied state and named questions, returning typed results. A yes/no question can return an estimated probability of yes. This is neither proof of truth nor a written explanation; preserve the evidence that a reviewer needs. [1][3]

Calibration still needs evaluation. Calibration describes how probabilities match observed outcomes across groups of predictions. It does not guarantee one answer. Test reliability on the documents, languages and risk categories you actually use. [5]

Batch independent questions. Checks sharing the same supplied state can travel together. They do not consume one another’s answers within that request. In the architecture below, code first filters passages; a later check receives the actual retained set. [2]

1–2. Choose the route and screen the boundaries.

Match the task to an approved handler. A lookup, ambiguous request and specialist review need different paths. Evaluate routing by the task and its consequences; an expensive or larger model is not automatically correct. [4]

Screen at both ends. Separate checks can flag sensitive input or unsupported output. A prompt-injection estimate can be useful, but it cannot grant access or replace isolation, limited privileges and approval. [10][11]

Open the complete SVG diagram (opens in a new tab)

Read the diagram as text

1. A business request

   Source evidence. Question, permitted context and a known user scope.

2. 1 · 2 · Independent semantic checks

   Model estimate. Which task? Which hazards? Is clarification needed?

3. Apply the approved policy

   Code policy. Enforce scope, approved handlers and cost limits.

4. Proceed or seek review

   Code policy. Choose an allowed handler, clarify or decline the task.

Connections and alternatives

• A business request → Independent semantic checks
• Independent semantic checks → Apply the approved policy
• Apply the approved policy → Proceed or seek review

4 → 3. Filter the evidence, then ask whether it is enough.

Retrieve a broad, bounded shortlist. Hybrid search combines keyword and vector results; rank fusion can merge their lists. Exact identifiers and paraphrases may both matter. Retrieval still sets the ceiling: no reranker can recover a missing candidate. [6][8]

Ask precise candidate questions. Is a passage useful evidence, a contradiction, or merely related? Keep source identifiers and conflicts visible. Application code applies the evaluated filtering and ranking policy; similarity and yes probabilities are different quantities. [7]

Check what survived. Once code has retained the passages, ask whether that set supports every required part of the answer. A check on the original shortlist cannot establish sufficiency of a different subset. Missing evidence leads to a bounded retry, clarification or an honest no-answer response. [2]

Open the complete SVG diagram (opens in a new tab)

Read the diagram as text

1. Hybrid retrieval shortlist

   Source evidence. Merge keyword and vector ranks; preserve source access.

2. 4 · Judge each candidate

   Model estimate. Estimate relevance, evidence and conflicting statements.

3. Filter in application code

   Code policy. Use versioned thresholds to form the actual kept set.

4. 3 · Check this set afterwards

   Model estimate. Does the retained evidence cover all required parts?

5. Draft with kept passages

   LLM draft. Evidence sufficient? Produce a source-linked draft.

6. Retrieve, clarify or abstain

   Code policy. No evidence? Use bounded retries or return for review.

Connections and alternatives

• Hybrid retrieval shortlist → Judge each candidate
• Judge each candidate → Filter in application code
• Filter in application code → Check this set afterwards
• Check this set afterwards → Draft with kept passages (Yes)
• Check this set afterwards → Retrieve, clarify or abstain (No)
• Retrieve, clarify or abstain → Hybrid retrieval shortlist

Illustrative P(usable evidence). C1: 0.92; C2: 0.74; C3: 0.18. Example estimates, not measured. At threshold 0.80, retain C1; exclude C2/C3. Evaluate your own policy.

5–6. Verify support before a draft becomes an action.

A citation needs to support its claim. Finding a source link is not enough. Check exact quotations mechanically, then assess whether the cited passage supports, contradicts or fails to address the draft proposition. Disputes and missing context need review. [9]

Permission is a separate decision. A proposed action can be semantically plausible yet unauthorized. The application validates identity, access, arguments and required approval before execution. A positive model score never authorizes a payment, email or record change. [11]

Open the complete SVG diagram (opens in a new tab)

Read the diagram as text

1. A draft claim or action

   LLM draft. Keep the claim, source ID and proposed tool arguments.

2. 5 · Does the source support it?

   Model estimate. Support, contradiction and absence need different routes.

3. 6 · Enforce action authority

   Code policy. Check source IDs, permissions, arguments and limits.

4. Approve, ask or deny

   Human review. Required approval precedes action; verify its result.

Connections and alternatives

• A draft claim or action → Does the source support it?
• Does the source support it? → Enforce action authority
• Enforce action authority → Approve, ask or deny

7–8. Keep the loop finite and the context accountable.

“Done” needs acceptance criteria. An assistant may believe it has finished while a required part is missing. Compare results with explicit criteria. Hard iteration, elapsed-time and cost ceilings stop work regardless of a model’s desire to continue. [13]

Select context without erasing obligations. Removing irrelevant or stale material can reduce clutter. Preserve required policy, source provenance, unresolved conflicts and necessary task state. Context selection is not permission to delete audit records or silently alter retention policy. [12]

Open the complete SVG diagram (opens in a new tab)

Read the diagram as text

1. 7 · Compare with acceptance criteria

   Model estimate. Does the result satisfy the specific requested task?

2. Enforce the hard budget

   Code policy. Attempts, elapsed time and cost cannot be extended by a score.

3. 8 · Keep the necessary context

   Code policy. Preserve policy and source IDs; remove stale duplicates.

4. Try the next bounded step

   Source evidence. Retrieve or revise only when policy still permits it.

5. Finish or report unresolved work

   Human review. Complete tasks return; exhausted budgets go to review.

Connections and alternatives

• Compare with acceptance criteria → Enforce the hard budget
• Enforce the hard budget → Keep the necessary context (Retry)
• Keep the necessary context → Try the next bounded step
• Try the next bounded step → Compare with acceptance criteria
• Enforce the hard budget → Finish or report unresolved work (Stop)

9. Observe more traces, and keep testing the judge.

Coverage and accuracy answer different questions. Running a check on every recorded trace can broaden monitoring coverage. It does not prove that every verdict is correct or every failure is covered. Keep human-labelled examples, inspect disagreements and report unavailable or failed evaluations. [14]

Make changes comparable. Version the model, question wording, source set and threshold policy. Measure false approvals, missed evidence, abstentions, reviewer effort and end-to-end cost and latency against the current workflow. Test on held-out cases before changing production policy.

Tracing is evidence of execution. Opik can record TypeSafe calls and evaluation results. The external patchy631/jev-as-judge repository demonstrates a tutorial approach; it is its author’s work, not a GTC deployment. A recorded experiment is not a quality certificate. [15][19]

Open the complete SVG diagram (opens in a new tab)

Read the diagram as text

1. A permitted, minimized trace

   Source evidence. Retain only evidence the evaluation is authorized to use.

2. 9 · Apply the versioned rubric

   Model estimate. Estimate support, request coverage and action honesty.

3. Compare with human labels

   Human review. Review errors, difficult cases and distribution changes.

4. Track quality and overhead

   Trace evaluation. Measure quality, cost and latency before expanding coverage.

Connections and alternatives

• A permitted, minimized trace → Apply the versioned rubric
• Apply the versioned rubric → Compare with human labels
• Compare with human labels → Track quality and overhead

Three business workflows worth testing.

Start with an accountable decision. These are illustrative pilot ideas, not completed client projects or measured savings. Each needs authorized sources, a process owner and a clear point where the system must stop.

Turn the architecture into a bounded pilot.

Agree the decision before selecting the model. Choose one recurring task and the mistakes that matter. A pilot should reveal whether a decision layer improves that workflow enough to justify operating it, including the cost of review and failures.

Scope the data boundary. A hosted Jev integration sends selected state to a provider. Agree permitted data, hosting, access, retention and contractual terms before connecting records. Nothing in this article activates that integration. [16]

• One task, a named owner and approved English/Arabic requirements.
• A representative labelled set, including missing, conflicting and hostile evidence.
• A baseline and held-out tests with acceptance criteria for consequential errors.
• Versioned questions, models and policy, with explicit review and no-answer routes.
• An access design, operating limits, monitoring owner and rollback plan.