As a Principal Data Engineer, you will combine deep technical expertise with engineering leadership to drive the design, development and evolution of our cloud-scale data platform. You will provide technical leadership across distributed data processing, data products, streaming architectures and data platform capabilities, while remaining hands‑on with coding, design and engineering best practices.
You will play a key role in shaping engineering standards, mentoring teams, influencing technical direction and delivering robust, scalable data solutions that enable our Engineers, Data Scientists, Analysts and Governance teams to unlock business value.
What You’ll Do
Lead the design, development and continuous evolution of cloud‑scale data platforms and data products across batch and streaming workloads
Act as the technical lead for complex data initiatives, providing guidance on solution design, engineering standards and implementation approaches
Design, build and optimise distributed data processing pipelines using PySpark, Python, Airflow and AWS services
Drive engineering excellence through code reviews, technical coaching, design reviews and adoption of software engineering best practices
Partner closely with Product, Architecture, Cyber Security, Data Science, Analytics and Governance teams to deliver scalable and reusable data capabilities
Champion modern data engineering patterns including Data Lakehouse architectures, ELT frameworks, Data Products and event‑driven processing
Influence technology choices, engineering standards and platform roadmaps while collaborating with Architecture and Enterprise Technology teams
Improve platform reliability, scalability, observability and operational excellence through automation, monitoring and continuous improvement
Drive adoption of CI/CD, Infrastructure‑as‑Code, testing strategies and engineering quality standards across the Data Engineering function
Mentor and develop engineers, fostering a culture of technical excellence, continuous learning and innovation
Support the adoption of AI and ML platform capabilities by building trusted, scalable and governed data foundations
Contribute hands‑on to the delivery of critical solutions, helping teams solve complex technical challenges and accelerate execution
What You’ll Bring
Significant experience as a Lead Data Engineer, Principal Data Engineer or Technical Lead within a large‑scale, data‑driven organisation
Strong hands‑on software engineering expertise with Python, PySpark and Apache Airflow
Deep experience designing and building distributed data processing systems and large‑scale data platforms on AWS
Strong knowledge of modern AWS data technologies including Glue, EMR, Lambda, DynamoDB, S3, EventBridge, Step Functions and related services
Proven experience building Data Lakehouse platforms, Data Products and ELT/ETL frameworks supporting analytics, ML and AI workloads
Expertise in streaming data architectures, event‑driven systems and real‑time data processing patterns
Strong understanding of data modelling, data quality, metadata management, governance and data platform best practices
Experience implementing software engineering best practices including CI/CD, automated testing, infrastructure‑as‑code and observability
Demonstrated ability to lead technical delivery across multiple teams and influence engineering direction without direct authority
Excellent stakeholder management and communication skills, with the ability to engage effectively across engineering, product, architecture and senior leadership audiences
Practical experience supporting Machine Learning and Generative AI platforms through scalable data engineering solutions
Passion for mentoring engineers and building high‑performing technical teams
Strong problem‑solving skills with the ability to balance strategic thinking and hands‑on execution
Experience operating within regulated industries, ideally Financial Services
#J-18808-Ljbffr…