Data Architecture and Modern Platform Design

Data architecture defines how an organization collects, organizes, transforms, protects and delivers data. It connects business requirements with the sources, models, flows and systems that make information usable for reporting, applications and AI.

A useful architecture makes practical questions answerable. Where does a figure come from? How recently was it updated? Who is allowed to see it? What happens when a load fails? How much work is required to introduce a new source or change a definition?

This guide explains the components of a modern data architecture, the differences between warehouses, lakes and lakehouses, and the roles of Medallion, data mesh and data fabric. It also sets out a process for choosing platforms and modernizing an existing environment without losing control of cost or operations.

What does data architecture need to achieve?

Data architecture translates the needs of information users into a coherent design. It includes logical decisions, such as the meaning of entities and relationships, and physical decisions, such as where processing runs and how data is stored.

For example, a business comparing product margins across countries needs more than a connection to several finance systems. The architecture must support compatible product identifiers, agreed currency conversion, consistent cost definitions and the history needed to interpret changes.

IBM’s explanation of data architecture places data structures, flows and management within an overall framework. In a business project, three related decisions have distinct purposes:

Strategy: which outcomes matter and which initiatives have priority.
Architecture: how data and capabilities should be organized to support those priorities.
Implementation: how the chosen services are configured, developed and operated.

An enterprise data strategy provides the priorities against which architectural trade-offs can be assessed. A daily management report and a fraud intervention have different needs for freshness, availability and response time.

What makes an architecture flexible?

Flexibility is the ability to accommodate a change without unnecessary rebuilding. It depends on clear interfaces, understandable definitions and visible dependencies. Adding more products can create more coordination work if those foundations are missing.

A practical test is whether a source change can be traced to the affected models and consumers. Another is whether the same business rule can be reused without being rewritten in every report. The benefits of modern cloud data architecture should be assessed against such tasks and their operating costs.

The components of a data architecture

The journey from a source to a decision usually involves several logical capabilities. They do not all need to be purchased or operated as separate services.

Sources: operational databases, business applications, files, documents and events.
Ingestion and integration: collection, change capture and coordination of incoming data.
Storage: retention of records, files, history and versions for defined purposes.
Preparation and modeling: validation, transformation and organization for consumption.
Access and consumption: queries, semantic models, reports, APIs and analytical or AI applications.
Operations and controls: monitoring, security, quality, lineage, recovery and cost management across the journey.

The Microsoft reference for big data architectures illustrates logical components and processing paths. A reference design helps identify requirements; it is not a requirement to implement every component in every project.

Batch processing, streaming and transformation

Batch processing handles groups of records on a schedule. Streaming processes events as they arrive, within the latency and delivery conditions supported by the design. The business decision determines the useful frequency. Faster updates can require more capacity, monitoring and treatment of duplicates or late events.

ETL transforms data before loading it into the intended destination. ELT loads it before transformation in that environment. Real systems can combine both. The choice depends on controls, available engines, data volumes and the needs of consumers.

OLTP supports operational transactions, while OLAP concerns analytical exploration. They describe processing purposes rather than competing pipeline tools. The explanation of ETL, OLAP and OLTP in data warehousing develops how these roles connect.

The article on ETL and data warehouse automation in the cloud develops the delivery perspective. Automation can reduce repeated work, but generated pipelines still require testing, ownership and maintenance.

The downloadable ebook on 11 best practices for data integration offers a companion for reviewing the connections between sources and the analytical platform.

Data warehouse, data mart, data lake and lakehouse

These concepts describe different ways to organize and serve information. They are not mutually exclusive choices for an entire company. One architecture can combine them for different workloads.

Data warehouse

A data warehouse organizes integrated information for analysis, reporting and historical comparison. It commonly supports business measures and SQL-based access. The supported formats, engines and processing options depend on the platform.

It can be a suitable approach when teams need shared definitions and repeatable analysis across several sources. It does not mean that every enterprise record must be copied into one database. The introduction to data warehouse concepts and use cases develops the fundamentals.

An operational database generally prioritizes transactions supporting an application. An analytical warehouse organizes information for queries across records and time. The comparison of a data warehouse and an operational database examines that distinction. Separating workloads may help protect operational systems, but the appropriate design depends on the load and product capabilities.

Data mart

A data mart provides information for a department, domain or analytical purpose. It may consume integrated enterprise data or use other sources according to its design. A smaller scope can simplify delivery, provided that shared measures remain compatible.

The comparison of data warehouse and data mart definitions and differences explains their scope and relationship. Departmental delivery should not create several conflicting definitions of the same business result.

Data lake

A data lake stores information in a range of file formats, including data that does not yet have a defined consumption model. It can support exploration of documents, logs or sensor records alongside structured data.

Flexible storage still needs ownership, metadata, permissions and retention rules. Keeping an original file does not make it understandable or suitable for every future use. The article comparing data lakes and data warehouses develops the storage and preparation differences.

Data lakehouse

A lakehouse combines lake-based storage with table management and analytical capabilities. Its behavior depends on the table technologies, query engines, catalog and integrations selected. It can allow engineering, BI and data science to share parts of the same foundation.

A lakehouse does not automatically remove duplicates, establish quality or cost less than a warehouse. It still needs modeling, access controls and operating practices. The guide to data lakehouse architecture and components explains the pattern, while the comparison of warehouse, lake and lakehouse in the cloud considers their different uses.

How should these approaches be compared?

A useful comparison starts with representative data and queries. Relevant criteria include formats, update frequency, simultaneous users, historical requirements, permissions and the team’s ability to operate the solution. Storage cost alone leaves out important differences in processing and maintenance.

The four steps for designing an enterprise data warehouse provide a more specific design sequence. Whether a warehouse is needed for a particular BI project depends on its integration, history and workload requirements; the relationship between data warehousing and business intelligence explores that planning question.

Medallion architecture and progressive data preparation

Medallion architecture organizes data preparation into logical layers, usually called bronze, silver and gold. The aim is to distinguish information received from sources, information that has passed agreed preparation rules and information ready for particular consumers.

Bronze: retains incoming records and source context, with the information needed for investigation and, where designed, reprocessing.
Silver: applies validation, normalization, duplicate handling and other agreed rules.
Gold: presents business models, measures or other outputs prepared for defined uses.

Databricks documents Medallion as a recommended design pattern, rather than a requirement. Layer names alone do not establish quality or transactional behavior. Those properties depend on the controls and storage technology actually implemented.

Each transition needs acceptance conditions and a failure path. Invalid references may be quarantined for review instead of silently discarded. Retention and reprocessing policies also need to prevent the layers from becoming indefinite copies without a purpose.

The detailed article on Medallion architecture and its bronze, silver and gold layers develops the pattern. Bismart’s downloadable Lakehouse Medallion architecture case study describes its application to a retail and ecommerce data environment. It provides a concrete reference for discussing source integration, preparation and business consumption; its reported results belong to that project.

Data mesh, data fabric and data as a product

These approaches address organization and coordination as well as technology. They should not be treated as replacements for a warehouse or a lakehouse: they answer different questions.

Data mesh and domain ownership

Data mesh gives business domains responsibility for data products within shared rules and platform capabilities. Its adoption requires teams able to maintain their products and coordinate across domains. Distributing ownership without that capacity can distribute unresolved problems.

The article on implementing decentralized data management with data mesh develops the organizational requirements. A domain might use a Medallion pattern internally while participating in a wider mesh.

Data fabric and coordination across systems

Data fabric is an approach to connecting and managing distributed data through capabilities such as integration, metadata and governance. It seeks more consistent discovery and use across systems, without implying that all data must physically move to one repository.

The introduction to data fabric and distributed data integration develops those capabilities. The comparison of data fabric and data mesh explains how technical coordination and domain accountability can complement each other.

Data products with explicit service expectations

Treating data as a product means defining its audience, meaning, access conditions, quality expectations and support. A dataset becomes reusable when another team can understand and rely on it without reconstructing the original processing logic.

Data fabric is also distinct from Microsoft Fabric. The first is an architectural approach; the second is a named Microsoft platform. Similar wording does not make their scope or requirements equivalent.

The ebook on the new era of data management provides a broader companion on management, governance and cloud environments.

Cloud, hybrid and multicloud data architecture

Cloud services can provide flexible capacity and managed capabilities, but the architecture still needs to determine where processing occurs, how access is controlled and what happens when a dependency fails. A migration can preserve existing complexity if only the location changes.

A hybrid design combines on-premises and cloud environments. A multicloud design uses services from more than one cloud provider. Either can be appropriate for specific constraints, existing investments or business requirements, while introducing additional integration and operating work.

The discussion of hybrid and multicloud investment develops the options. The article on financial mistakes in cloud infrastructure adds an economic perspective. Architecture evaluation should include data transfer, duplicated processing, monitoring, support and the cost of keeping old and new environments running together.

Data residency requirements also affect the design. Relevant questions include where records, backups and logs are stored, who can access them, and how cross-border processing is controlled. These requirements need specialist validation for the organization’s circumstances rather than assumptions based solely on a region label.

How to choose a data platform

A platform provides services that implement an architecture. Microsoft Fabric, Azure Synapse, Databricks and Snowflake offer different combinations of capabilities. Their relevance depends on the workload, existing environment, skills and operating requirements.

A practical evaluation uses the same representative scenario for each candidate. It checks ingestion, transformations, query behavior, access controls, monitoring, recovery and cost under realistic conditions. An isolated demonstration of query speed does not establish the total effort of running the service.

The article on Azure Synapse in a cloud data architecture explains its architectural role. The comparison of Azure Synapse and Azure Databricks provides background for evaluating different processing needs. Product features and commercial conditions should be checked against current documentation before a decision.

For the Microsoft ecosystem, the introduction to Microsoft Fabric and transition planning develops the platform perspective. The article on OneLake in Microsoft Fabric examines the storage layer. Logical reuse can reduce some copying, but it does not remove the need to assess permissions, supported access paths and data movement.

The downloadable Microsoft Fabric guide covering features, licensing and migration supports the initial evaluation. Pricing, eligibility and migration requirements remain matters for a current assessment.

Performance, reliability and operating cost

Performance problems can originate in source access, transformation logic, storage layout, models, queries or concurrent demand. More compute may reduce a symptom without addressing repeated scans, unnecessary movement or an inappropriate model.

Useful operational measures include time from source change to usable information, query response under representative concurrency, failed or retried loads, recovery time and cost per workload. Each needs a clear owner and an agreed response when it moves outside acceptable conditions.

The article on SQL syntax and query readability provides introductory background. Query tuning requires additional checks against the actual engine and execution plan; general rules about table width or query layout are not reliable substitutes.

Automated compute management is another consideration. The overview of adaptive warehousing and compute allocation introduces the idea. Snowflake’s current Adaptive Compute documentation describes a specific compute service with edition and regional conditions. A product capability should not be confused with a universal storage architecture or a guarantee of lower cost.

Reliability also requires tested recovery. A pipeline should detect failures, preserve enough context for investigation and support an agreed recovery process without introducing inconsistent results. Successful scheduled runs alone do not demonstrate that recovery works.

How to design and modernize a data architecture

A design process works from a defined use to the requirements and dependencies needed to support it. A bounded first release can test the whole journey before the organization expands it.

Definition of the use and consumers. The team establishes the decision, application or process and the owner who accepts the result.
Mapping of the current environment. The baseline records sources, transformations, duplicate logic, dependencies and responsibilities.
Agreement on requirements. These cover freshness, quality, access, retention, availability, recovery and cost.
Design of the data journey. The team compares patterns and components and documents why the chosen option fits.
Testing with a representative workload. Tests check functional results, operating behavior and failures.
Preparation for operations. Responsibilities cover monitoring, support, change management and incident response.
Expansion based on evidence. Validated capabilities are reused, with decisions reviewed when conditions change.

A short decision record can capture the problem, alternatives, choice, rationale and conditions that would trigger a review. It helps prevent a temporary constraint from becoming an unexplained permanent rule.

Modernization also includes retirement. Old interfaces, duplicate models and unused copies continue to create cost and risk until they have a clear removal plan. Changes can be introduced gradually while continuity and reconciliation are verified.

Data architecture for AI

An AI-ready architecture needs to account for authorized sources, context, freshness and the way outputs will be evaluated. Documents and other unstructured content matter when they form part of the intended application.

For a system that retrieves information to answer questions, the design should identify permitted sources, preserve access boundaries and specify how changed or withdrawn content is reflected in derived indexes. Permission to store a document does not automatically authorize every application to use its contents.

Where an agent can take action, the design also needs limits on that action, traceable execution and a way to intervene or recover. Evaluation should examine the intended task: whether answers are supported, permissions are respected and missing information is handled appropriately.

The guide to preparing enterprise data for AI develops these foundations. The architecture should follow the agreed conditions of use before individual AI components are selected.

An architecture example with acceptance criteria

In this hypothetical example, a company with several ERP systems needs to compare margin by product and region. The design begins with definitions of revenue, cost and product mapping, together with the update frequency required for management decisions.

The team designs ingestion, controlled preparation and a shared consumption model. A Medallion pattern could organize the preparation. A warehouse with staging and analytical models could also address the requirement. The choice depends on the expected workloads, existing systems and operating capacity.

Acceptance criteria provide evidence that the design is working:

Amounts reconcile with the sources for the agreed period and scope.
Unmatched references are detected and assigned for resolution.
Information arrives within the agreed decision window.
Permissions reflect the responsibilities of different users.
A failed load can be investigated and recovered through a tested procedure.
Response time and cost have been measured under representative demand.

The example describes a design exercise rather than a client result. Its value is making requirements testable. When a need changes, the team can identify which agreement, component or flow must be revisited.

Common questions about data architecture

What is the difference between data architecture and a data platform?

Data architecture defines the organization of information, flows and capabilities. A data platform supplies services used to implement that design. One platform can support different architectures, and one architecture can span several platforms.

Is a lakehouse better than a data warehouse?

Neither is universally better. The appropriate choice depends on formats, queries, skills, access requirements and total operating cost. A lakehouse still needs modeling and controls, while a warehouse can coexist with other processing and storage approaches.

Is Medallion architecture the same as data mesh?

No. Medallion organizes progressive data preparation into layers. Data mesh distributes responsibility across domains with shared rules and capabilities. A domain can use Medallion inside a wider data mesh arrangement.

Are data fabric and Microsoft Fabric the same thing?

No. Data fabric is an approach to managing and connecting distributed data. Microsoft Fabric is a specific platform. The similar name does not establish that their purposes, capabilities or implementation requirements are identical.

Does modernization require moving everything to the cloud?

No. Modernization can improve models, integration, controls and operations across existing environments. What moves depends on requirements, dependencies, continuity needs and the cost of migration and coexistence.

How can a business assess whether its architecture works?

It can test whether usable data reaches its consumers under agreed conditions for quality, freshness, access, recovery and cost. The effort needed to introduce a change is another practical indicator of the design’s flexibility.

From architectural choices to a usable data platform

A reliable data architecture makes the full journey understandable, from the source record to the business use. It makes ownership, dependencies and operating requirements visible enough to test and maintain.

The Lakehouse Medallion case study shows a concrete application of these principles. For the delivery perspective, Bismart’s approach to data warehouse implementation explains how integrated information supports business analysis.

Where fragmented sources or unreliable preparation are blocking a project, Bismart’s data platform design and implementation services connect source integration, architectural decisions and operating controls.