Data Engineer

Location:
Job Type: Contract
Discipline: Technology
Reference: 1180128
Posted: 12 days ago
A contract opportunity exists for an experienced Data Engineer to join a key project improving the findability, accessibility, interoperability, and reusability of scientific data. Towards accomplishing this, the project is implementing automated data processing pipelines using Azure cloud. Ultimately this will form the foundation and lead to the formation of a Qld Science Data Platform of shareable and re-useable raw and curated science data, and upon which more extensive machine learning can be applied for greater environmental insights.

To assist achieving this, the Department is recruiting an experienced Data Engineer for an initial contract to 30 June 2022 with possible extensions to join the project team. This role will report to the project's Solution Architect and perform the duties applying their skills and experience as follows.

These duties can be performed using a combination of WfH and onsite.

Duties
  • Automate common, repeatable, and tedious data preparation and integration tasks to minimize manual processes and improve productivity for scientists.
  • Design, build, test and maintain scalable Azure Data Factory and Azure IoT APIs from a variety of structured and unstructured data sources.
  • Monitor, optimize, & maintain existing data pipelines at scale.
  • Resolve defects and issues and build automated tests to prevent reoccurrence.
  • Support production deployments and conduct handover activities to BAU support.
  • Support the planning and implementation of data platform services including infrastructure-as-code via Terraform, and tracking data consumption patterns.
  • Collaborate with scientist stakeholders across varied departments within and outside the organization including those engaged in Open Source projects.
  • Train scientists and science data analysts in data preparation techniques and configuring data processing pipelines to self-help and consume data for their own use cases.
  • Promote the available data and analytics capabilities and expertise to science leaders and educate them in leveraging these capabilities in achieving their science goals.
Experience and Skills
  • 4+ years development experience with Azure Data Components or similar ETL tools such as SSIS - Azure Data Factory, Azure SQL, Azure Data Lake, Azure Analysis Services, Azure Synapse and Azure Databricks, Azure Machine Learning.
  • Database experience in MS SQL Server, Data Warehouse, blob storage, NoSQL/Hadoop oriented databases.
  • Experience working with big data sets and file formats including Parquet, Spark, AzureML.
  • Experience with a range of diverse data sets including structured, non-structured, CSV, XML, JSON, Web API
  • Experience with popular data discovery and analytics tools like PowerBI, Tableau
  • Experience applying Continuous Integration, Continuous Delivery and Agile practices, unit & integration tests, and development practices.
  • Experience working in an Agile environment using tools such as Jira and Confluence
  • Azure Data Engineer Associate Certification is desirable
  • Development experience in Python, R, PowerShell is desirable
  • Experience in API development for analytic/ML end points is desirable
  • Data Governance experience including Master Data Management (MDM) and Data Quality tools and processes is desirable
The role closes Wednesday 24th November 2021.