Insights25 March 2026Updated 26 October 20265 min read

AI Procurement: Three Questions Beyond 'AI-Powered'

Three vertical neon lights in blue, pink and green illuminate a dark background with a translucent rectangular panel.

We recommend starting an AI procurement review with three questions: where does the answer come from, what happens when it is wrong, and where does the data go? Treat these as requests for evidence, rather than a questionnaire that a supplier can satisfy with reassuring wording. Ask to see the behaviour in the proposed deployment before agreeing that the feature is ready for your team.

These are our recommended questions, not a finding about what every buyer asks. The UK government's AI procurement guidance advises teams to investigate model limitations and the origin of supplier training data, and to evaluate testing across different conditions. Use that approach to define what you need to establish, then give each unresolved point an owner before purchase.

Question 1: where does the answer come from?

A bright orb with fine traceable threads of light running back to a few distinct source blocks

Ask the supplier to distinguish information retrieved from your records from material generated without a retrieved source. For a factual answer about your organisation, request a reference to the specific record and revision used. Then open that reference and check whether it supports the answer, rather than accepting the presence of a link as the test.

This distinction matters because NIST's Generative AI Profile identifies confidently presented false content as a risk and warns that generated citations can themselves be false. Our recommendation is to test the supporting passage and the answer together. Include questions whose answer is missing from the approved records, and ask the supplier to demonstrate how the system handles that absence.

For a proposed trial, provide a small set of approved documents containing an old policy and its replacement. Ask which version the system uses, how that choice is made and what the user sees when the documents disagree. Agree the expected result with the person who owns the policy before running the test. Record a failure if the system presents a superseded instruction as current without making the conflict visible.

The architecture discussion in Erder, Pureur and Woods' Continuous Architecture in Practice, chapter 8, treats data provenance, metadata management and observation of machine learning performance as system concerns. We would therefore ask who maintains the source collection and how a correction reaches the deployed product. Request evidence that a user cannot retrieve material outside their permissions, including through a generated summary. Keep that access test separate from checking whether the answer is factually correct.

Question 2: what happens when the model is wrong?

Ask for a demonstration of failure handling, with the expected outcome written down before the demonstration starts. Include an unsupported answer, a source that contradicts the proposed answer and an unavailable dependency. For each case, specify whether the product should stop, ask for clarification or send the work to a named person.

Do not accept a confidence score as the acceptance criterion on its own. Ask what the score measures, how it was evaluated on the intended task and which action follows a low score. If the supplier cannot demonstrate that relationship, keep the score out of decisions about unattended operation.

NIST's AI Risk Management Framework includes response, recovery and communication in risk treatment. It is a voluntary framework, not a certificate that the proposed product is safe. We recommend turning those responsibilities into a written operating procedure for your deployment. Name the person who receives a reported error, the evidence they need and the action they can take to stop further use.

For outputs that trigger a consequential action, require an explicit approval step during the trial. Test what happens when the reviewer rejects the output or does not respond. Do not count a review control as passed merely because an approval button exists. Require a demonstration that the downstream action cannot proceed through another route without the agreed approval.

Set acceptance criteria for the task you intend to buy, including unacceptable failures as well as successful outputs. The government's evaluation guidance specifically includes defining acceptable model performance and testing under a range of conditions. Our recommendation is to retain the test inputs, expected results and observed failures so that a later change can be checked against the same requirements.

Question 3: where does the data go?

Streams of light flow inside a drawn boundary ring, with one stream passing out through a gate on its edge

Request a data flow diagram for the exact deployment being offered, rather than accepting a general statement about local hosting or a cloud region. Ask the supplier to show where prompts and uploaded files are processed, where outputs are stored and what is copied into operational logs. Include backups, support access and any separate services used to prepare documents or produce answers in that review.

For every destination, request the operator, location, purpose, retention period and permitted access. Ask separately whether customer material is used for training or service improvement, and put the agreed restrictions in the contract. Require evidence for the configuration you will receive, rather than treating a supplier's available privacy option as an enabled control.

We recommend assessing local and hosted deployments against the same written requirements. For a local proposal, ask the team to demonstrate the permitted network connections and account for any outbound traffic during the trial. For a hosted proposal, request equivalent evidence about the services involved and the people able to access the data. Do not treat the location of the model alone as sufficient evidence to approve the whole deployment.

Have your security and privacy specialists assess the proposed flows against your organisation's obligations before using sensitive records in the trial. This is a procurement recommendation, not a claim that one hosting location establishes legal compliance. Where the supplier cannot establish a destination or retention rule, record it as unresolved rather than assuming the most favourable answer.

Put the answers into acceptance criteria

We recommend carrying the evidence from these questions into the purchase decision and delivery plan. For each requirement, record the demonstration, the result and the person authorised to accept any remaining risk. Keep unresolved controls visible, with an agreed restriction on use until the missing evidence is supplied. Ask for re-evaluation when the supplier changes the model, retrieval process or data flow.

Apply the same discipline to your own product description: explain the task, the limits and the controls you can demonstrate. We would remove a promise that cannot be traced to evidence rather than replace it with a more persuasive adjective.

Update, October 2026

Our engineering article What the Agent Does Not Know to Look For describes supplying agents with dated provenance and explicitly identified gaps before work begins. For procurement, we recommend asking to inspect that initial context alongside the final answer.

Sources

Want this in your team?

Tell us what you're building and where you want to get to, and we'll think it through with you.

Let's have a chat Follow by RSS →

All insights →