Data Engineer - Databricks, Unity Catalog (4-month daily rate contract)
Key Purpose of the Role
The Data Engineer will design, build, and maintain the technical foundations that support the Data Risk and Governance Program. This role focuses on developing automated data quality and profiling frameworks, building robust data pipelines, and integrating governance tooling across EnergyAustralia's cloud platforms. The engineer will ensure data assets are secure, observable, and aligned with governance policies enabling confident investment and operational decisions across the program.Key Responsibilities
• Design, build, and enhance serverless and container-based data quality and profiling frameworks on AWS using Sodacore.• Develop automated pipelines for data validation, classification, and observability across Redshift, Databricks, and other platforms.
• Work with cloud-based platforms such as Databricks and Amazon Redshift to access, transform, and analyse large datasets.
• Utilise APIs to extract and integrate data from various systems and sources.
• Integrate data quality outputs into reporting and cataloguing tools such as Power BI and Alation.
• Implement and support third-party SaaS data quality agents within cloud infrastructure and CI/CD pipelines.
• Ensure secure handling of sensitive and classified data, including access control auditing and compliance with data governance policies.
• Collaborate closely with team members and stakeholders to align solutions with business needs and technical standards.
• Produce clear, user-friendly documentation including requirements, designs, and support materials.
• Contribute to a culture of continuous improvement, knowledge sharing, and operational excellence.
Capabilities, Qualifications & Experience
• Hands-on experience designing and maintaining data pipelines and quality frameworks in cloud environments, preferably AWS.• Proficiency with cloud data platforms including Databricks and Amazon Redshift.
• Experience integrating data cataloguing and reporting tools such as Alation and Power BI.
• Familiarity with CI/CD pipelines and embedding third-party SaaS tools within cloud infrastructure.
• Solid understanding of data governance principles, including access control, data classification, and compliance.
• Strong API integration skills for extracting and connecting data across multiple systems and sources.
• Experience working in cross-functional teams and agile delivery environments.
• Tertiary qualifications in Computer Science, Information Technology, Data Engineering, or a related discipline.
• Excellent written and verbal communication skills, with the ability to produce clear technical documentation for varied audiences
