Formula-1 Data Analysis with Azure Databricks
- Built a real world data project using Azure Databricks and Spark Core - Formula1 Data Analysis.
- Used Azure Databricks, Delta Lake, Spark Core, Azure Data Lake Gen2 and Azure Data Factory (ADF).
- Created notebooks, dashboards, clusters, cluster pools and jobs in Azure Databricks.
- Ingested and transformed data using PySpark in Azure Databricks.
- Transformed and analysed data using Spark SQL in Azure Databricks
- Implemented a Lakehouse architecture using Delta Lake.
- Created Azure Data Factory pipelines to execute Databricks notebooks.
- Created dashboards using databricks notebooks.
- Created Azure Data Factory triggers to schedule pipelines as well as monitor them.
- Connected to Azure Databricks from PowerBI to create reports.
- Gained a comprehensive understanding about Unity Catalog and the data governance capabilities offered by Unity Catalog.
- Implemented a data governance solution using Unity Catalog enabled Databricks workspace.
