Data Hygiene: Make Business Data Fit for Its Purpose

Data hygiene starts with the decision the data must support. The Government Data Quality Framework defines quality in terms of fitness for purpose, rather than cleaning alone. Our recommendation is to make that purpose explicit before choosing a cleansing tool or commissioning another dashboard.

For a field service team, start with a practical question: can the dispatcher trust this record enough to send someone to the right asset? For a finance team, ask whether an invoice refers to the intended customer and approved work. These are suggested starting points, not evidence that every organisation has the same defects.

Decide what good enough means

The framework separates completeness, uniqueness, consistency, timeliness, validity and accuracy. Those distinctions are useful when writing an acceptance test. A populated field is not proof that its value is correct.

Consider an illustrative inspection record with an asset identifier, completion time and outcome. We would specify which fields must exist, which identifiers are permitted and who can confirm the outcome. We would also ask how old the record may be before the intended decision needs a fresh check.

Write the rule in operational language first. For example: an inspection cannot be marked complete without an outcome, but a draft can be saved with an explanation of what remains missing. Have the inspection owner approve that distinction before an engineer turns it into validation logic.

Avoid a single target for every field. Decide which missing values should stop a particular action, which should create a review task and which can remain unknown. Record the reason alongside the rule, so the next person changing the form can see what the check protects.

Give each rule an owner and a test

The government's action planning guidance starts with critical data, quality rules and an initial assessment. We would apply that sequence to one business process before extending it across an organisation.

Choose a process owner who can answer questions about meaning, not merely somebody with database access. Ask them to approve examples of an acceptable record, an unacceptable record and a record that needs judgement. Keep these examples with the rule.

For the first assessment, choose a defined population and record the selection criteria. Include the assessment date, the rule version and the unresolved exceptions. When comparing later results, use the same definition or explain why it changed. Do not present a narrower sample as an improvement in the whole system.

We would separate format checks from factual confirmation. In the inspection example, checking that an asset identifier has the expected structure should be a different test from confirming that the inspection refers to that asset. Assign the second check to a person or reference record with the authority to resolve it.

Correct the cause, not only the spreadsheet

The framework's guidance distinguishes root cause analysis from repeatedly treating symptoms. Use that distinction when deciding what a cleansing task must deliver.

Suppose an import assigns an unknown inspection type to a default category. Treat this as a hypothetical defect to investigate. Examine the import mapping before asking a team to edit the resulting rows. Agree whether unknown values should be rejected, held for review or accepted with an explicit warning.

Before applying a bulk correction, keep the original values and define the proposed transformation. Test it on a reviewable copy. Ask the process owner to inspect ambiguous cases rather than treating a similar name as sufficient evidence to merge records.

For deduplication, define what counts as the same entity and what evidence permits a merge. Decide which record will remain authoritative and how references to the retired record will be handled. Keep the correction history, including who approved it. Do not make deletion the default response to a suspected duplicate.

Apply the same discipline to offline work

For an app intended to work without a connection, we recommend separate acceptance tests for local entry and server reconciliation. Treat these as design requirements to prove in the app, not as a claim that a particular framework provides them automatically.

At local entry, check the rules that can be evaluated from the information available on the device. Show the user which checks remain pending. In the inspection example, allow a saved draft to say that its asset reference still needs confirmation rather than displaying an unqualified success message.

At reconciliation, test what should happen when the server has a changed asset record, an incompatible category or another submission for the same inspection. Agree who resolves each case and what the person sees while it remains unresolved. Preserve the submitted values until the resolution has been reviewed.

Include a replayed submission and an interrupted transfer in the test plan. Ask the team to demonstrate the final record and its correction history, not just that a network request succeeded. Keep the data quality rule separate from the mechanism that moves the record between devices.

Make quality visible to the next user

The guidance recommends documenting assessments and explaining data quality to users. For each dataset we would record its intended use, accountable owner, assessment date and known limitations.

Start the next review with unresolved exceptions from the previous one. Choose a review interval based on the decision and how often the relevant data changes. Assign each corrective action an owner and a way to demonstrate that the original defect no longer appears in the assessed population.

For a practical first step, pick one field that determines whether work proceeds. Write its rule, gather disputed examples and agree how a correction will be recorded. Use that small exercise to decide what needs a software change and what needs a clearer operational decision.

Update, October 2026

For teams applying these ideas to agent context, our later engineering article on grounding packs and dated evidence describes how provenance and explicit knowledge gaps are carried into an agent's starting context. We recommend reviewing the quality of those inputs separately from the model's output.

Sources

• Government Data Quality Hub, "The Government Data Quality Framework", 3 December 2020.
• Government Data Quality Hub, "The Government Data Quality Framework: guidance", 3 December 2020.