Data Engineering Lead

Requirements

• We’re looking for a technical, high-ownership Data Engineering Lead who can ship, operate and improve production systems — and translate platform work into real product and customer outcomes
• Hands-on Data Engineering Leader: You’ve owned production data platforms end-to-end and can lead technical direction without stepping away from the code
• Dagster expertise: Demonstrable experience building asset-based orchestration (assets, sensors, IO managers, resources) in production
• Warehouse & SQL depth: Advanced BigQuery skills — partitioning, clustering, materialised views, slot/cost management and query optimisation
• Ingestion reality: You’ve operated Airbyte or similar, built resilient API/FTP/SFTP ingestion patterns, and handled rate limits, pagination, partial failures and schema evolution
• Production API & feeds experience: You’ve shipped customer-consumable APIs or feeds (SLAs/contracts, monitoring, provenance) and can design stable downstream contracts and rollout plans
• Python & engineering rigor: You write clean, testable Python (typing/pydantic where appropriate), author unit and integration tests, and enforce CI and linting standards
• Multi-cloud & IaC: Comfortable with AWS (ECS/Fargate, RDS, S3) and GCP (BigQuery, GCS), and production Terraform experience
• Operational excellence: You’ve built observability, incident response, runbooks and reduced on-call toil through automation and better design
• Clear communicator & coach: You can explain trade-offs simply, unblock teams fast, and raise engineering quality through calm, direct leadership
• If you’re excited about this role but your experience doesn’t perfectly align with the job description, we encourage you to apply anyway

What the job involves

• As our Data Engineering Lead, you’ll own the technical roadmap for our data platform and raise the bar on reliability, scalability, and engineering quality
• This is a hands‑on, player/coach role
• Own the Data Platform Roadmap: Set technical direction and deliver the highest‑leverage platform improvements across reliability, cost, developer experience, and scale
• You’ll choose, prioritise and deliver the next 12 months of platform work
• Deliver customer‑facing APIs & Feeds: Support the design and delivery of API‑backed feeds and enrichment pipelines that become product features and revenue streams
• Level Up Orchestration in Dagster: Build and refine asset‑based pipelines, sensors, schedules, backfills, IO managers and monitoring patterns that are robust, idempotent and easy to operate
• Define standard patterns for incremental jobs, full refreshes and reverse ETL
• Make ingestion boring (in the best way): Improve and scale ingestion across Airbyte OSS and DLT: handle schema drift, connector health, rate limits, retries, checkpointing and operational resilience so pipelines run without heroics
• Strengthen BigQuery foundations: Own and evolve our best practices, partitioning & clustering strategy, slot/cost management and query performance guardrails
• Raise data quality & observability: Implement freshness SLOs, automated checks, validation, provenance and alerting so the business and customers can trust the data. Ship runbooks, incident playbooks and automated remediation where possible
• Enable customer pipelines & reverse ETL: Own correctness, availability and SLAs for customer‑facing workflows, including schema contracts and safe rollouts
• Infrastructure as Code & CI: Own Terraform modules, CI/CD flows for both infra and data code, and deployment safety gates that prevent costly mistakes
• Coach and grow the team: line‑manage and mentor engineers, raise standards through code reviews and testing, and build a high‑performance engineering culture that values operational excellence

#J-18808-Ljbffr…