- Design, build, and maintain scalable data pipelines and ETL processes using Azure Data Factory for data ingestion from diverse sources including on-premises databases (Authority), API-based CAMMS, and data files in CSV and JSON formats.
- Utilize Azure Data Lake Storage (Gen2) and Delta Lake for managing data in different stages (Bronze, Silver, Gold).
- Implement data processing workflows using Azure Databricks to transform raw data into processed high-quality data assets.
- Develop and optimize data models in Databricks and Power BI ensuring they meet performance and scalability requirements.
- Enable advanced analytics capabilities by providing clean, structured, and reliable data for analysis and reporting using Azure Databricks and Power BI.
- Implement and manage DevOps practices for continuous integration and continuous deployment (CI/CD) to enhance data engineering workflows through Git integration.
- Establish and enforce data standards, policies, and procedures including a data quality framework to ensure the accuracy, completeness, and consistency of data.
- Support the Data and Insights teams by providing efficient data access and processing tools on Azure.
- Monitor, support, and improve the technology landscape, including maintenance and cost savings.
- Assist the Team Leader with remediation activities identified from internal and external audits.
- Bachelor’s degree or above in Computer Science, Data Engineering, Information Systems, or a related field.
- 5+ years of experience in data engineering, data modelling, data management, and data governance using Azure cloud services.
- Strong expertise in data architecture, ETL processes, and data modelling on Azure.
- Proficiency in programming languages such as Python and SQL with demonstratable experience in Azure Data Factory, Azure Databricks, and other Azure data services.
- Experience with DevOps practices, including CI/CD pipelines.
- Experience in ingesting data from on-premises RDBMS databases, APIs, and various formats like CSV, XLS and JSON.
- Data governance in Azure Purview and Unity Catalog.
- Strong analytical and problem-solving skills with the ability to translate business requirements into technical solutions.
- Excellent communication and collaboration skills to work effectively with cross-functional teams.