Data Mesh is one of the newest self-service platform concepts to address data architecture and data management needs, while enabling data accessibility and availability. The model aims to directly connect data producers, data consumers and data owners, building trust in the quality of the key asset - data as a product. This blog looks at addressing the challenges organization face when implementing Data Mesh.

Evolution of data architecture

Data architecture defines the blueprint for data processing mechanisms and processes and describes how insights for end-user functions are generated from data storage. There have been a variety of data architecture approaches throughout the history of data storage and operations in the context of modern IT and the ever-evolving business requirements.

While the data warehouses were among the first centralized architectural approaches, it was not long before business intelligence tools emerged in the 90s, such as decentralized data marts capable of interpreting information in graphical summary, thus expanding the user base to executives. At the beginning of the millennium, we saw a move towards data lakes, a model which prioritizes large storage volumes of any type of data and cost efficiency over performance.

More recently, self-service infrastructure platforms have gained attention. These present a decentralized approach with the premise of allowing non-technical users to access and integrate data from different sources without relying on IT or data specialists (who can be scarce and costly resources). The implementation of the self-service infrastructure platform architecture requires addressing two main aspects - the technology stack and the organizational change and governance.

Data Mesh overview and advantages

Data Mesh is a rather new concept, seeking primarily to provide a blueprint to address - in an optimized way - the organizational change and governance aspect. It aims to solve data management within organizations and is designed to support agile, decentralized, and scalable data architecture. It targets the pitfalls of the traditional centralized data architecture, such as silos and rigid structures, by enabling more distributed ownership and governance of data and promoting a culture of data collaboration and experimentation across the organization. Data Mesh is defined by four key principles:

1. Data as a product
2. Domain ownership
3. Computational federated governance
4. Self-service platform.
   

Within the concept of Data Mesh, data should be organized into self-contained domain-specific data products, which are owned and governed by domain teams. To support this decentralized approach, Data Mesh also advocates a set of technical and organizational capabilities that enable teams to collaborate and exchange data across the organization in a scalable and flexible way.

Key solutions for Data Mesh implementation challenges

Implementing Data Mesh requires a significant shift in the way organizations understand and manage their data. This involves addressing challenges in the following areas:

1. Organizational culture: Traditional data management approaches rely on a centralized data governance model, where a single team or department manages all data-related activities. In contrast, Data Mesh requires domain teams to take ownership of their data products, which can be a significant cultural change for many organizations.
   
   

   Addressing this challenge requires an organization-wide shift in mindset, with a focus on fostering a culture of data collaboration and trust. To support this change, organizations need to introduce adequate incentive mechanisms to ensure that those who act in the way that supports those objectives, feel valued. Domain teams must be empowered to take ownership of their data products, and they must be given the necessary tools and resources to build, test, and deploy them independently.

2. Domain data ownership: Domain teams take ownership of their data products while defining the scope and boundaries of their domain, as well as taking responsibility for managing and maintaining their data products. This can be a daunting task for teams that are used to relying on centralized data management teams.
   

   To address this challenge, organizations must establish clear governance policies, ensuring compliance with data privacy laws and regulations, including data quality metrics and standards for each domain.

3. Data product lifecycle management: Domain teams manage the entire data product lifecycle, including development, testing, deployment, and retirement. This can be a challenging task for many domain teams, as they may lack the necessary expertise and resources to manage these activities effectively. Organizations need to ensure that domain teams have access to the necessary training, tools and infrastructure to build, test, and deploy their data products independently.
4. Data integration: Each domain team must ensure that their data products can be integrated with other data products across domains. This requires a significant amount of collaboration and communication between domain teams, which can be challenging for organizations with a highly siloed culture. Solving this challenge will require organizations to establish clear policies and procedures for data integration, including data standards, documentation and formats.
5. Data security and compliance: Domain teams’ ownership of data products can create security and compliance risks. Each domain team must ensure that their data products comply with data privacy laws and regulations and that they are secure from unauthorized access. To remain secure and compliant, firms must have clear policies in place as well as providing domain teams with the necessary training and resources to ensure that their data products comply with these policies and procedures.
   
   

Conclusion

Financial services institutions are becoming more focused on getting value from their data. However, they are increasingly confronted with complex data universes, lack of domain knowledge, disappointingly low flexibility in creating new capabilities and long time-to-market before new data-centric functions are ready.

Data Mesh, as one of the newest approaches amongst self-service platforms, offers an effective solution to data architecture and data management needs, through increased accessibility and availability of data and greater trust in the quality of the data. 

Understanding that the Data Mesh concept is more than just a technology stack, highlights a significant shift in organizational thinking and culture, which are prerequisites for this concept. Data Mesh opens distributed data sets, accelerating access to data, improving data delivery accuracy and building cross-organizational trust, while leveraging data-oriented self-service design. Self-service further simplifies access to data, breaks down silos and enables sharing of live data at scale.

By addressing the challenges outlined above and providing domain teams with the required tools, resources, and training, organizations can efficiently assess the suitability of the Data Mesh model to ensure successful implementation.

Identifying which model is suitable for your organization and implementing Data Mesh can be a challenging task. Capco has extensive know-how in enterprise-wide data strategy programs and data maturity assessments. Contact us to discuss how we can help your firm establish, improve or scale data and achieve your transformational goals.