Financial institutions require solutions for creating and sharing high quality data sets, internally and externally for analytics and AI/ML.

In this presentation we will introduce one of the foundational pieces of OS-Climate Data Commons: its data platform that provides its constituents - producers and consumers of data - with a toolbox for sharing datasets, as well as easily creating and sharing new datasets on the platform.

Bringing concepts such as Data-as-Code, GitOps, and CI/CD to data ingestion, transformation and publishing, and based on the conceptual framework of the Data Mesh, we will present and demonstrate a Data Platform built with open source components such as Apache Airflow, Spark, Apache Iceberg, Trino, Unity and more and running cloud-natively on Kubernetes. This provides solutions for a wide range of problems faced by services institutions in the data space - from self-servicing data infrastructure for running ETL pipelines, to decentralizing data product ownership while preserving security and regulatory compliance in a federated governance model.

Come see it at work in a practical exercise!