Insights23 January 2026Updated 25 October 20265 min read

Why Gut Feeling Needs Testing: Data-Driven AI in Enterprise Operations

A glowing digital tunnel with neon lines and squares, symbolising Business Intelligence data pathways.

Gut feeling is not always wrong, and replacing it with AI does not remove the need to test a decision. Daniel Kahneman and Gary Klein's research on intuitive expertise distinguishes genuine skill from confidence by asking whether the environment contains useful patterns and whether the person has had opportunities to learn them. For enterprise operations, we recommend making that distinction before choosing what to automate.

The useful question is whether a proposed decision process performs better on the work it will actually support. Compare expert judgement, an explicit rule and a model against the same defined outcome. Do not assume that a recommendation becomes reliable because software produced it.

Keep the expertise, test the judgement

Kahneman and Klein conclude that subjective confidence is an unreliable guide to the validity of intuition. Their argument does not dismiss experience. It asks whether the person has learned relevant cues in a sufficiently predictable environment.

For a hypothetical field service team, start with a recurring decision such as which delayed job needs attention first. Ask experienced planners to describe specific cases, the information they considered and what happened afterwards. Keep disagreements in the record rather than turning the most senior person's preference into the correct answer.

Define the outcome separately from the previous decision. For this example, use whether the job breached its agreed deadline, alongside the time spent reviewing the queue. Do not label every historical allocation as correct simply because a planner approved it. Treat the planners' explanations as candidate rules to test, not proof that a model should reproduce their choices.

Make the inputs fit that decision

Before training a model, write down the meaning and origin of each input. For the job queue, specify how deadlines, job status, required skills and engineer availability are recorded. Distinguish the time an event happened from the time its record reached the central system. Give an owner responsibility for resolving missing or contradictory records.

This is our proposed preparation checklist, rather than a guarantee that clean data will produce accurate predictions. NIST's AI Risk Management Framework treats validity and reliability in relation to intended use and calls for realistic test sets representing expected operating conditions. For this trial, retain examples of incomplete records and unusual jobs so those conditions are tested too.

Check legacy imports against the same definitions before using them in the evaluation. For example, do not combine a legacy status meaning “engineer attended” with a new status meaning “customer accepted completion”. That is an illustrative mismatch to investigate, not a report about a particular client's system. Keep the source values and document the transformation so a reviewer can reproduce the input.

Separate policy from prediction

For this workflow, keep approved operational constraints outside the model's discretion. A requirement for a particular qualification should be checked as an explicit condition, rather than inferred from who previously received similar jobs. Ask the operational owner to approve the condition and its exceptions.

Use a decision tree if the agreed conditions can be expressed as branches a planner can review. Reserve a learned model for a separate prediction, such as estimating the chance that a job will miss its deadline. Treat its output as a proposal for investigation, not authority to bypass an eligibility condition.

This separation is our design recommendation, not a claim that every decision tree is understandable or every model needs vast training data. Give reviewers the applicable rule version, the inputs and the reason a case was referred to them. Record an override and its reason so the team can inspect where the rule or prediction was unhelpful.

Test the complete workflow against a baseline

Before the trial, agree which errors matter and what improvement would justify using the new process. Compare the model-assisted queue with the existing planning process and a simple rule, such as ordering eligible jobs by deadline. Use the same cases and information available at the point of decision. Keep later completion records out of the inputs used to predict those completions.

NIST recommends documenting test methods and allows results to be broken down by data segment. For this trial, inspect results by job type and by whether records were complete, instead of relying only on an overall score. Count unnecessary escalations as well as missed deadlines. Include the planner's checking time in the comparison.

David Farley describes tests as small experiments that predict and verify intended behaviour in chapter 14 of Modern Software Engineering. Apply that approach to the workflow: state the expected response before running each case. Include a case where the correct response is to request more information rather than rank a job.

Define what happens when information is unavailable

For any offline version of this workflow, specify which decisions may use locally held records and which must wait for confirmation. Show when the records were last refreshed and identify missing inputs beside the recommendation. If engineer availability cannot be confirmed, make confirmation the next task rather than silently treating the last known value as current.

Do not make continuous learning on the device a default requirement. Start the trial with an approved model and rule version, then review proposed changes before deployment. Sculley and colleagues describe unstable data dependencies and versioned inputs, including the maintenance costs of keeping versions. Our recommendation is to make those dependencies visible before adding local adaptation.

Keep checking after deployment

Sculley and colleagues also warn that a model can influence its own future training data. For this hypothetical queue, review which jobs received attention and which did not, rather than examining only the cases the model prioritised. Do not treat a planner's acceptance of a suggestion as evidence that the job outcome improved.

NIST describes ongoing testing or monitoring as a way to assess deployed validity and reliability. Name the person who will review errors and the conditions that will pause automated recommendations. At Dendro Logic, we recommend starting that conversation with one decision, its evidence and its acceptance test. Let's have a chat about which operational decision your team wants to test first.

Update, October 2026

For a later engineering application of explicit context and visible gaps, see What the Agent Does Not Know to Look For. It describes grounding packs that carry provenance, dates and explicit unknowns.

Sources

Want this in your team?

Tell us what you're building and where you want to get to, and we'll think it through with you.

Let's have a chat Follow by RSS →

All insights →