How to choose an AI pilot you can actually evaluate
Start with one recurring decision, a usable sample and an agreed definition of success. A smaller pilot can reveal more than an ambitious demo with no clear test.
Published guide
1. Choose a workflow, not a technology
“Use AI in finance” is too broad to test. “Draft an explanation for a monthly cost variance, with links to the underlying records, for a controller to approve” identifies an input, an output and a responsible person. Start with a recurring task whose current process you can observe. Record who performs it, how long review takes and what happens when information is missing.
Keep the first boundary narrow: one business unit, one document type or one reporting period. Do not begin by granting an agent access to every system. A useful first pilot may recommend a next action while leaving all changes to a person.
2. Assemble a representative sample
Ask the process owner for examples that they are authorised to use. Include ordinary work, exceptions and incomplete records. Separate a development sample from a held-out evaluation sample so the team does not keep improving against the same familiar examples. Record source dates, ownership, access permissions and any redactions.
For bilingual workflows, include realistic English and Arabic examples and agree who can judge each language. A translated interface alone does not demonstrate that the system understands the underlying business documents.
3. Agree what passing means
Choose thresholds with the business owner before building. Compare the proposed workflow with the current process using the same examples. An impressive answer is insufficient if checking it takes longer than doing the work directly.
For the cost-variance example, a passing answer identifies the periods, reproduces the ledger difference and links to the records used. If the meter reading is missing, it can still report the recorded cost change but must leave its energy explanation unresolved. The reviewer should be able to distinguish a correct calculation from an unsupported explanation.
- Accuracy: does the output answer the intended question, with supporting records?
- Abstention: does it clearly flag missing, conflicting or insufficient evidence?
- Review effort: can the responsible person check and correct the result efficiently?
- Access: does each user see only the information they are allowed to use?
- Recovery: can the team return to the existing process after a failure?
4. Test the uncomfortable cases
Try a stale policy, a duplicated invoice, an ambiguous request and a document containing misleading instructions. Record the expected behaviour before running each test. A system that confidently invents a source, exposes restricted material or silently overwrites an approved value should fail that test. Keep human approval before actions such as payments, external messages or changes to business records.
Log the outcome and the reason for each review decision without collecting unnecessary sensitive data. Retest failures after a change and check that previously correct examples still work.
5. Decide whether to buy, build or stop
Test an existing product first when it already fits the workflow, permissions and integration needs. Consider a custom build when a specific integration or decision process provides material value that the available tools cannot support. Include licences, integration, review time, maintenance and an exit path in the comparison.
End the pilot with an evidence pack: tested scope, results, unresolved failures, a named operational owner and a recommendation. Continue only when the benefits justify the remaining work. “Improve the data first” or “keep this manual” can be the right conclusion.