Data Quality: Dimensions, Assessment and Improvement

Data quality describes how well information meets the requirements of its intended use. Accuracy, completeness, consistency, validity, uniqueness and timeliness provide different ways to assess it. Their importance depends on the process, the people using the information and the consequences of an error.

A successful data load does not establish that its contents are correct. A populated customer field may contain the wrong identifier. A report may reconcile with a source that uses a different definition of revenue. Improving quality means making these conditions explicit, testing them and resolving problems throughout the data lifecycle.

This guide covers data quality assessment, cleansing, validation rules, observability and ongoing improvement. It explains how business and technical teams can establish whether information is fit for reporting, operational processes and AI, with specialist articles and downloadable resources for each stage.

What does data quality mean in business?

Data is fit for purpose when it supports a defined task under known conditions. A monthly sales trend may tolerate a delay that would make the same information unsuitable for checking stock before accepting an order. Quality therefore needs a business context, rather than a universal label attached to a dataset.

The starting point is the decision or process that depends on the information: what must be known, when it is needed and what could happen if it is wrong. The article on why data quality matters to business operations explores the connection with rework, analytical reliability and confidence in reporting.

Reliable information is only part of a sound decision. Its interpretation, relevance and limitations also matter. The discussion of data quality and data-driven decisions develops this relationship. More records do not automatically improve an analysis, and a clean dataset cannot establish causality on its own.

Data governance and data quality

Data governance establishes ownership and authority: who agrees definitions, approves changes and accepts exceptions. Data quality management turns expectations into measurable checks and corrective work. The specialist explanation of how data governance and data quality connect provides further context.

Data integration creates the flows in which many of those controls operate. A transformation can standardise a value while changing its meaning. Quality checks need to follow the journey from the source through preparation to the final business use.

The six data quality dimensions

The UK Government Data Quality Framework identifies six core dimensions. Each addresses a different question; a strong result in one does not establish a strong result in the others.

Accuracy: whether a value represents the real event or entity correctly.
Completeness: whether the records and fields needed for the agreed scope are present.
Validity: whether values satisfy specified formats, types, ranges or permitted categories.
Consistency: whether representations that should agree do so within the defined context.
Uniqueness: whether an entity or event is represented without unintended duplication.
Timeliness: whether information is available sufficiently soon for its intended use.

Why dimensions need separate checks

A delivery date can pass a format check and still record the wrong day. A required field can be filled with a placeholder that makes it complete but inaccurate. An order identifier can legitimately appear more than once when each row represents a different order line.

These distinctions affect the rule being tested. Uniqueness may require a combination of order number and line number. Completeness may need a comparison with the source population, because checking populated fields cannot reveal orders that never arrived.

Where dimensions conflict, the trade-off should be documented. An early provisional view may be useful if it clearly identifies missing records; a financial reconciliation may require a later, complete population. A single overall score can conceal a failure that makes one particular use unacceptable.

Data integrity and quality checks

Integrity can refer to database relationships, protection against alteration or the preservation of information through processing. Each control needs a precise definition. Preventing unauthorised changes does not prove that the original value was accurate.

The article on data integrity and verification methods introduces related controls. Referential checks, reconciliation and change protection answer different questions and should be evaluated separately.

How is data quality assessed?

A data quality assessment compares a defined population with the requirements of a specific use. It records the rules applied, the coverage of testing, the exceptions found and the effect of unresolved issues on the intended task.

Data profiling supports this work by examining missing values, distributions, patterns and frequencies. It helps identify questions worth investigating. An unusual distribution is a signal, not proof of an error: a sudden increase in orders may reflect a promotion rather than a faulty pipeline.

Define the scope. Identify the source, period, population, row-level meaning and consumer.
Agree the definitions. Establish what each field represents and when an absence is legitimate.
Profile the data. Examine ranges, categories, null values, repetitions and relationships.
Validate representative cases. Compare selected records with sources or people able to confirm the underlying event.
Separate findings. Distinguish confirmed defects, acceptable exceptions and cases that cannot yet be assessed.
Prioritise by consequence. Connect each issue with the affected task, records and business exposure.

Sampling needs its own explanation: how records were selected, which situations were covered and what conclusions are justified. A sample check should not be described as validation of every record.

The downloadable Data Maturity Model guide places this assessment within the organisation’s wider capabilities. It is a companion for prioritising improvements, rather than a dataset validation tool.

Data cleansing and preparation

Data cleansing applies agreed corrections to identified defects. Data preparation organises information for a subsequent task. Both need enough documentation to explain what changed, why it changed and whether the result remains suitable for its consumers.

Standardisation: reconcile date conventions, units and codes before converting values.
Duplicate handling: define the entity and the evidence needed to merge or remove records.
Missing values: distinguish unknown, not collected and not applicable.
Outliers: investigate unusual observations before excluding them.
Reference matching: compare identifiers and categories with maintained reference records.

Replacing an unknown amount with zero can distort an average. Removing an exceptional transaction can erase a genuine event that needs investigation. The appropriate treatment depends on meaning and consequences, and its effect should be tested before the result is reused.

The article on data wrangling in Azure provides a preparation example in a marketing data context. Its product references and configuration steps require comparison with the current environment before implementation.

Fixing the source of recurring errors

A downstream correction may be necessary to keep work moving, but repeated cleansing can hide the cause. If an application accepts an invalid customer category, cleaning every export leaves the capture problem in place.

Where a source change is not feasible, the workaround needs an owner, an explanation and a review date. Consumers also need to know which values were corrected or inferred and which limitations remain.

Data quality rules and validation

A data quality rule translates an expectation into a testable condition. A useful specification identifies the applicable records, the condition being evaluated, the frequency, the responsible person and the action required when the check fails.

Checks can address schema, mandatory fields, permitted categories, relationships, business logic or reconciliation. Examples include confirming that a customer reference exists, that a required delivery date is populated and that totals match a source under the same period and scope.

Data contracts between producers and consumers help document these expectations and the handling of changes. A contract becomes operational when the corresponding checks and notification procedures are implemented.

Measuring quality without hiding coverage

For an applicable rule that can be evaluated, a pass rate can be calculated as records passing the rule divided by applicable records evaluated, multiplied by 100. The result should be reported alongside the population, exclusions and records that could not be evaluated.

If a required field is present in 980 of 1,000 applicable records, its completeness is 98%. This describes the calculation, not an acceptable threshold. It also says nothing about whether the 980 values are accurate.

A failed assessment job is different from a failed business rule. Where evidence is unavailable, the result should remain unavailable or not assessed. Turning that situation into a zero or an automatic pass creates misleading certainty.

Rules themselves need testing with valid cases, known defects and legitimate exceptions. After a transformation changes, a previously sensible control may reject valid information or approve an incorrect result.

Testing, monitoring and observability

Testing checks defined expectations. Monitoring follows selected signals over time. Data observability combines those signals with context about pipelines and dependencies to support investigation. Their coverage and capabilities depend on the implementation; none provides universal visibility by definition.

The introduction to data observability across the data pipeline develops signals such as freshness, schema changes and volume variation. The article on data quality testing, monitoring and observability compares the approaches.

For example, fewer transactions in a report could result from a source outage, a filter change or a genuine decline in sales. A useful alert connects the observed variation with upstream jobs, recent changes and affected consumers, so a responsible team can investigate.

Making alerts actionable

Each alert needs an intended recipient and a decision to support. Expected patterns should account for weekends, seasonal demand and planned source changes. Repeated false alarms can reduce attention to the alerts that matter.

Operational review should examine missed incidents as well as noisy notifications. The relevant question is whether a problem was detected and resolved before, or with less impact on, the business task that depends on the data.

Incident response and service objectives

A data incident needs a defined scope, a responsible person and evidence of recovery. The response should establish which consumers are affected, whether use can continue and how limitations will be communicated.

Contain the impact. Identify affected outputs and determine whether publication needs to pause or carry a warning.
Investigate the cause. Follow sources, transformations and recent changes.
Correct and reprocess. Apply the agreed fix while controlling duplicates and unintended changes.
Validate recovery. Confirm that both the dataset and downstream outputs meet the relevant conditions.
Prevent recurrence. Update capture rules, tests, documentation or responsibilities.

Metadata and lineage in data integration support this investigation by making origins, transformations and dependencies easier to trace. The recorded lineage still needs to cover the actual systems involved.

Agreeing a service level for the data

A service indicator measures a condition, such as the proportion of reporting days with the required information available before an agreed time. A service objective establishes the target over a period. An SLA formalises commitments and associated conditions where appropriate.

Google’s guidance on implementing service level objectives provides a reliability approach that can inform data operations. Applying it to a dataset requires a consumer-relevant measure, a reporting window and an agreed response when the target is missed.

Data quality tools and automation

Data quality tools can support profiling, rule execution, documentation, alerts and investigation. Evaluation should use representative data and tasks, including a known defect, a legitimate exception and a recovery scenario.

The comparison needs to cover supported sources, rule versioning, access controls, understandable results, operational integration and maintenance costs. The ability to calculate a score does not establish that the right business conditions have been tested.

Microsoft Purview data profiling illustrates how exploration can inform checks. Its documentation on data quality rules in Unified Catalog describes rule-based evaluation. Supported sources, permissions and configuration need to be checked for the proposed deployment.

AI-assisted rule suggestions can provide a starting point, but someone familiar with the data and process still needs to evaluate their meaning. Automatically generated rules can overlook legitimate exceptions or formalise a misleading pattern.

Placing controls within the architecture

Some conditions can be checked at ingestion; others require multiple sources to be joined. Controls should be placed where the necessary evidence exists and before an incorrect result can affect a consumer.

Medallion architecture and its preparation layers provide one way to separate stages. A silver or gold label does not certify quality. Each transition needs defined criteria and checks that support the downstream use.

The downloadable Lakehouse Medallion architecture case study describes a retail and ecommerce application involving integration and data preparation. Its results belong to that project and should not be treated as forecasts for another implementation.

Data quality for AI

Quality requirements differ between training a model, evaluating its behaviour and retrieving documents to answer questions. The intended AI task determines what needs to be checked.

Training and evaluation data require attention to labels, coverage, collection conditions and availability at the time of prediction. Underrepresented situations can make an aggregate result look stronger than performance in an important subgroup.

Document retrieval introduces other checks: current versions, duplication, text extraction and the relationship between a fragment and its original source. A retired policy can remain in an index even after the source document has changed.

Clean input does not remove the need to test the model, permissions or resulting actions. A correct customer record does not authorise an agent to change it. The guide to preparing enterprise data for AI provides a broader companion on integration, quality, governance and AI use.

A worked example using order data

Illustrative example. A company needs to analyse delivered orders by customer. Each row represents an order line. The agreed checks cover customer references, delivery dates and duplicate lines.

Of 1,000 lines, 20 lack a customer identifier that should be present. Completeness for that field is 98%. The team examines the value and customers affected before deciding whether the data can support the analysis. Twenty missing lines may carry a disproportionate share of revenue.

Repeated order numbers initially look like duplicates. Investigation shows that several refer to different lines within the same order. The uniqueness rule is therefore applied to order number and line number together.

A source comparison also finds dates that satisfy the required format but represent dispatch rather than delivery. The preparation rule is corrected and downstream metrics are recalculated. The recorded outcome includes each check, its coverage, unresolved issues and the authorised use. It is not summarised as a universal quality score.

Building a data quality improvement programme

A practical programme begins with a material process and a manageable set of sources. The first delivery should establish the baseline, implement relevant controls and demonstrate whether an identified problem has improved.

Agree the consumers, scope and acceptance conditions.
Record the baseline and any assessment gaps.
Assign ownership of rules, exceptions and incident response.
Address recurring causes and test recovery.
Incorporate monitoring into normal operations.
Review defects, manual effort, delays and maintenance costs.

Business evaluation should compare both errors and the effort needed to handle them. Less reprocessing or reconciliation can be useful evidence; changes in sales or margin require consideration of other influences before being attributed to quality improvements.

The downloadable ebook on 11 best practices for data integration develops preparation and control considerations across the data flow.

Frequently asked questions about data quality

What are the main data quality dimensions?

Accuracy, completeness, validity, consistency, uniqueness and timeliness are commonly used. Each needs a definition and measure appropriate to the task. The dimensions should remain visible separately where a combined score would hide an important failure.

Does valid data always mean accurate data?

No. Validity establishes compliance with a defined rule, such as a date format. Accuracy concerns whether the value represents the correct real-world event. A correctly formatted date may still be the wrong date.

How does data cleansing differ from data quality management?

Cleansing corrects identified problems. Quality management also includes defining expectations, measuring results, preventing recurring defects and responding to incidents during use.

Does data observability replace data testing?

No. Testing evaluates defined conditions, while observability provides signals and context for investigation. Both require appropriate coverage and people able to act on the findings.

What is an acceptable data quality score?

There is no universal acceptable percentage. The answer depends on the rule, assessment coverage and consequences of the remaining errors. A rare failure can still affect a critical transaction.

Does good data quality establish GDPR compliance?

No. Accuracy is one element of the GDPR principles explained by the European Commission. Purpose, lawful processing, retention and protection also matter. Passing a quality check is not a compliance certification.

Downloadable resources for data quality improvement

Data Maturity Model guide for assessing organisational capabilities and improvement priorities.
11 Best Practices for Data Integration for reviewing preparation, consistency and controls across data flows.
Lakehouse Medallion architecture case study for a documented example of integrated preparation.
Guide to preparing enterprise data for AI for connecting quality with integration, governance and AI requirements.

From an assessment to ongoing controls

A useful first step is a defined dataset, a business requirement and an agreed response when a check fails. That creates a concrete basis for deciding which causes to address and where additional controls are justified.

For organisations evaluating automated validation, profiling and alerts, Bismart Data Quality Framework presents a solution to assess against those requirements. The evaluation should include the relevant sources, business rules and operational responsibilities.