Data Platform Engineer

Salary: £? – ? per year

Requirements

• We need strong AWS cloud platform migration and Terraform expertise.
• We need in-depth data engineering experience.
• We need experience migrating data platforms and data workloads from legacy Databricks environments to modern AWS cloud-based data platforms.
• We need strong Terraform and Infrastructure as Code (IaC) skills, including developing and maintaining Terraform code for data platform deployments.
• We need experience with Python, PySpark, SQL, data pipeline development, ETL/ELT design, data modelling, data quality, and data validation.
• We need experience with AWS services, including S3 and IAM.
• We need familiarity with CI/CD concepts, DevOps principles, and Agile delivery.
• We strongly prefer experience in Government, Public Sector, or related domains.
• We strongly prefer experience with Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE), or NHS/Healthcare data.
• We require candidates to be active SC Cleared or fully eligible to undergo the full SC Security Clearance process.

Responsibilities

• We will focus on data engineering, cloud platform migration, infrastructure automation, and supporting the delivery of large-scale data processing solutions.
• We will support a major data platform modernisation programme, including migration of data pipelines, notebooks, and data workloads from a legacy Databricks environment to modern AWS cloud-based data platforms.
• We will develop and maintain Terraform code to support data platform deployments.
• We will support deployment of data platform components, jobs, permissions, and environment configurations.
• We will analyse current data engineering solutions and identify data migration requirements.
• We will onboard datasets, users, and workloads onto the target platform.

Technologies

• AWS
• CI/CD
• Cloud
• Databricks
• DevOps
• ETL
• IAM
• Support
• Python
• PySpark
• SQL
• Security
• Terraform

This is a 6‑month rolling contract based in London, with a hybrid working model of 3‑4 days per week remote and 1‑2 days per week in the office, plus occasional travel to Leeds.

#J-18808-Ljbffr…