Role: Senior Azure Data Engineer (Databricks + GenAI)
Location: UK Remote
Engagement: Permanent
Overview:
We are seeking a Senior Azure Data Engineer to take ownership of the data engineering capability across multiple enterprise-scale Azure Databricks programmes. This role sits at the intersection of data engineering, cloud architecture, and GenAI enablement, building production-grade lakehouse platforms that power analytics, automation, and AI-driven applications. You will act as a technical lead and client-facing engineer, responsible for designing, delivering, and governing scalable Azure + Databricks data platforms.
Key Responsibilities:
1. Architecture Leadership
- Define Azure Databricks lakehouse architectures (medallion model)
- Review and approve engineering designs across projects
2. Data Engineering Delivery
- Build end-to-end pipelines using:
- Azure Databricks (PySpark, Delta Lake, Workflows)
- Azure Data Factory / Synapse Pipelines
- ADLS Gen2
- Deliver curated data products (bronze / silver / gold)
3. GenAI Enablement
- Prepare data for RAG and AI workloads
- Work with Azure OpenAI + Azure AI Search
- Implement embedding, chunking, and retrieval strategies
4. CI/CD & Infrastructure
- Implement IaC using Terraform or Bicep
- Manage CI/CD pipelines using Azure DevOps or GitHub Actions
- Use Databricks Asset Bundles for deployment automation
5. Governance & Security
- Implement Unity Catalog, Purview, RBAC, and data lineage
- Ensure secure-by-design architecture (Key Vault, Entra ID)
6. Client Engagement
- Lead workshops, design sessions, and technical discussions
- Translate business requirements into structured engineering delivery
7. Mentorship
- Guide and support mid-level engineers
- Enforce engineering standards and best practices
Required Experience:
- 8+ years in data engineering
- 4+ years hands-on Azure data platform experience
- Expert-level Azure Databricks (PySpark, Delta Lake, Jobs, tuning)
- Strong Azure Data Factory / ADLS Gen2 / Azure SQL experience
- Solid Python and SQL engineering background
- Experience with lakehouse / medallion architecture
- CI/CD & IaC (Terraform or Bicep + Azure DevOps/GitHub Actions)
- Understanding of Azure AI Search / Azure OpenAI / RAG concepts
- Experience with Unity Catalog / Purview / data governance frameworks
Soft Skills:
- Strong client-facing communication skills
- Confident leading technical design discussions
- Ability to manage ambiguity and define structure
- Mentoring and leadership capability
Nice to Have:
- Power BI / Fabric experience
- Salesforce data integration experience
- Microsoft certifications (DP-203, DP-700, AZ-305, etc.)
- Experience in real-time or operational data environments
- Exposure to Data/AI Centre of Excellence environments