Senior Data Engineer

1w1 week ago

AMP

AU · Full-time · A$180,000 – A$250,000

About this role

AMP is a familiar name if you live in Australia or New Zealand. We're evolving into a nimbler business with new leadership amid changing society. Help people create their tomorrow in banking, super, retirement, and finances through upturns, downturns, and life transitions.

Every day, we enable people to see and make more of their financial potential, building on over 170 years of service. Engineer, operate, and improve enterprise-scale data platforms and pipelines on AWS for secure, scalable analytics. Design end-to-end data pipelines translating business needs into technical designs across the full SDLC.

Take end-to-end ownership of deliverables with minimal supervision, driving from requirements to production. Develop big data solutions using Apache Spark, Python, and SQL for batch and streaming workloads. Build pipelines with AWS stack including S3, Glue, EMR, Athena, and Redshift for performance and cost efficiency.

Apply AI-driven practices for code generation, testing, and architecture optimization in data platforms. Embed data governance, security, observability, and CI/CD for reliable operations. Join big thinkers redefining financial services and turning legacy into positive impact for Australia's prosperity.

Requirements

Extensive Scala/Spark expertise required; PySpark desirable
Proficiency in Python and SQL for large-scale data processing
Experience with AWS data stack: Amazon S3, AWS Glue, EMR, Athena, Amazon Redshift
Knowledge of AI-driven software engineering and design techniques for data pipelines
Skills in data governance controls including access management, data quality, and lifecycle policies
Expertise in secure data integration with IAM, encryption, and secrets management
Ability to implement observability practices: logging, metrics, monitoring, alerting, and dashboards

Responsibilities

Engineer, operate, and continuously improve enterprise-scale data platforms and data pipelines on AWS
Design and build end-to-end data pipelines across the full SDLC, translating business requirements into technical designs
Take end-to-end ownership of deliverables with minimal supervision, identifying risks and driving to production
Develop high-performance big data processing solutions using Apache Spark, Python, and SQL for batch and streaming workloads
Build, maintain, and optimise data pipelines using AWS data stack including Amazon S3, AWS Glue, EMR, Athena, and Amazon Redshift
Apply AI-driven software engineering practices including code generation, refactoring, testing, and troubleshooting
Design and automate batch and streaming data pipelines with resilient orchestration and error-handling
Establish CI/CD and automation practices for data pipelines and infrastructure using Git and infrastructure-as-code