A large global payment provider desired to be able to enable data collaboration and discovery between multiple internal teams utilizing the Databricks Platform. Rearc partnered with them to understand the vision and develop a data governance strategy which enable internal teams to manage their own data while meeting organization security goals.
Rearc utilized Unity Catalog and Databricks Workspaces to develop a hub-and-spoke model, the DataHub. Each internal team represents a spoke which is provisioned with a set of standardized user groups to provide workspace admin, end user and read-only roles.
These spoke teams are able to manage their own resources within catalogs in a self-service manner. To facilitate cross-team collaboration or data sharing, each spoke is able to add data sets to a catalog which is able to be accessed by other spokes. Access may be granted to an entire team or specific individuals. Spokes can discover metadata about shared data and request access to listed data sets to promote discovery.
The client was able to have a consistent governance model for internal teams and the processes to quickly create a new spoke within the DataHub model. Teams were able to collaborate and generate data sets to be broadly consumed within the Databricks platform by the organization. Moreover, the foundational governance and guard-rails from DataHub supported and enabled development of additional strategic projects within the data platform.
Read more about the latest and greatest work Rearc has been up to.
This project for a Fortune 500 company strengthened the security of Databricks-integrated data lake pipelines by implementing automated vulnerability detection, secure credential management, and third-party dependency scanning, resulting in a compliant and resilient infrastructure.
Rearc upskilled a Product Security team in public cloud infrastructure as code, developing a secure Terraform module for AWS SNS, and tracking remediation of Trusted Advisor findings.
As organizations strive to innovate rapidly, enhancing development velocity for their applications have become crucial. Rearc partnered with a Fortune 250 customer with the primary objective of streamlining the creation of net-new applications with built-in best practices. Key challenges included ensuring seamless integration with existing systems, maintaining robust security practices, accelerating time-to-market, retaining quality, and more.
Rearc designed and implemented a foundational governance pattern, DataHub, supporting sharing of data between internal buisness segments on Databricks.
Tell us more about your custom needs.
We’ll get back to you, really fast
Kick-off meeting
