Lead Data Engineer (AWS Data)

Requirements

• The successful candidate will bring deep expertise in data engineering, distributed data processing, and cloud-native platforms, with a strong focus on AWS-based data ecosystems
• Proven experience in data engineering and cloud-based platform delivery
• Strong understanding of distributed data processing and scalable system design
• Ability to lead delivery while remaining hands‑on technically
• Strong analytical, problem‑solving, and communication skills
• Experience working in client‑facing and delivery-focused environments
• Ability to mentor and develop engineering teams
• Strong hands‑on experience with:
• AWS cloud services, especially AWS Glue
• Python / PySpark for large‑scale data processing
• SQL for querying, transformation, and validation
• Configuration‑driven development (e.g., YAML)
• Experience building and operating:
• Data pipelines
• ETL/ELT workflows
• Cloud-native data platforms
• Familiarity with:
• Data lakes and Lakehouse concepts
• Distributed processing frameworks (e.g., Apache Spark)
• Strong understanding of:
• ETL vs ELT patterns
• Performance tuning and optimisation
• Experience with:
• Version control (Git)
• CI/CD and DevOps practices

What the job involves

• We are seeking an accomplished and detail‑oriented Lead Data Engineer – AWS to join our Data & AI practice
• This role is critical in designing, building, and optimising end‑to‑end data pipelines and platforms, enabling scalable data processing, advanced analytics, and AI‑driven solutions
• You will play a key role in ensuring data quality, integrity, performance, and reliability, supported by strong engineering and testing practices
• As a senior practitioner, you will collaborate with architects, engineers, and analysts to deliver secure, scalable, and high‑performing data solutions, leveraging technologies such as AWS Glue, Python/PySpark, SQL, and configuration‑driven frameworks (e.g. , YAML)
• You will thrive in a collaborative, client‑facing environment, with a passion for solving complex technical challenges, ensuring delivery excellence, and driving modernisation through cloud‑native engineering practices
• Act as a senior engineer within data engineering and cloud platform initiatives, supporting delivery across complex transformation programmes
• Collaborate with architects and stakeholders to define and implement scalable AWS‑based data solutions
• Contribute to solution design, estimation, and delivery planning
• Lead engineering workstreams and ensure high‑quality technical delivery
• Design, build, and optimise scalable data pipelines and data processing frameworks on AWS
• Develop and maintain ETL/ELT pipelines using:
• AWS Glue
• Python / PySpark
• SQL
• Configuration‑driven frameworks (e.g., YAML)
• Implement robust data ingestion, transformation, and processing patterns
• Build reusable data services, components, and frameworks
• Define and implement testing strategies for data pipelines, ensuring reliability and accuracy
• Validate data processing workflows using:
• Python / PySpark transformations
• SQL‑based validation logic
• Configuration‑driven orchestration
• Develop automated testing, monitoring, and alerting solutions
• Ensure:
• Data completeness
• Data accuracy
• Consistent transformation behaviour
• Drive improvements in observability and pipeline resilience
• Lead development on AWS services including:
• S3‑based data lakes
• Supporting services within the AWS data ecosystem
• Support implementation of modern data architectures, including data lakes and Lakehouse‑style platforms
• Optimise pipelines and jobs for performance, scalability, and cost efficiency
• Data Transformation & Modelling
• Define and implement data transformation logic aligned to business requirements
• Support data modelling approaches for analytics and platform use cases
• Ensure consistency, usability, and quality across data assets and pipelines
• Collaborate with:
• Solution Architects
• Data Engineers
• Analysts and ML engineers
• Provide technical leadership and mentoring to engineers within the team
• Promote engineering best practices, automation, and reusable solutions
• Contribute to engineering standards, documentation, and knowledge sharing
• Ensure data quality, integrity, and reliability across data platforms
• Implement and enforce secure coding and data handling practices
• Support compliance with:
• GDPR
• Regulated environment standards (where applicable)
• Contribute to monitoring, auditing, and operational processes

#J-18808-Ljbffr…