The Role
The Interconnectors Data Platform (ICDP) is a cloud-based data warehouse enabling critical commercial, financial modelling, and operational decision-making. As the platform moves toward a modernized Medallion Architecture and Azure-native ingestion patterns, we are hiring a Senior Data Engineer to provide architectural direction and hands-on technical leadership.
Your responsibilities:
Data Architecture & Platform Engineering
– Lead the design and implementation of scalable data architectures, including Bronze/Silver/Gold layered models.
– Drive the platform’s architectural roadmap and ensure alignment with modern engineering best practices.
– Develop robust, secure, and observable ingestion and transformation pipelines.
Pipeline Development & Operations
– Lead the migration from legacy ETL tools (e.g., Matillion) into modern Azure-based pipelines, including Azure Functions, Azure Data Factory (ADF), and event-driven frameworks.
– Build and maintain performant SQL-based transformations, curated layers, and reusable data models.
– Ensure CI/CD, testing, version control, and observability are fully embedded in workflows.
Data Quality & Governance
– Implement data validation, reconciliation, profiling, and auditability across platform layers.
– Work with business stakeholders to ensure output meets analytical and operational needs.
Leadership
– Mentor data engineers and contribute to the growth of technical capability within the ICDP team.
– Collaborate with Product teams, IT&D, and external partners to deliver high-quality outcomes.
– Serve as a technical authority for engineering approaches, patterns, and standards.
Your Profile
– Python: Strong, hands-on experience building production-grade data pipelines, transformations, and orchestration using Python.
– Advanced SQL: Expert-level analytical SQL, query optimization, and data modelling.
– Azure Cloud: Experience with Azure Functions, Azure Data Factory, Azure Storage, Key Vault, and cloud security fundamentals.
– Data Warehousing: Deep understanding of data architecture principles, dimensional modelling, and scalable enterprise data design.
– Strong grounding in Git, CI/CD, automated testing, branching strategies, and modern engineering practices.
– Experience with API ingestion, SFTP ingestion, and resilient pipeline design.
Soft Skills
– Strong problem-solving and architectural thinking.
– Excellent communication and stakeholder collaboration skills.
– Ability to provide clarity and leadership in complex or ambiguous technical landscapes.
Desirable skills/knowledge/experience:
Experience contributing to or leading data-platform re-architecture programs.
– Exposure to Medallion/Lakehouse patterns or Databricks-style ecosystems.
– Experience in regulated or high-assurance data environments.
– Collaboration with modelling teams or environments where engineering and modelling intersect.
…