Role Summary
Lead Software Data Engineer responsible for building scalable data systems and pipelines in a modern AWS environment, collaborating with product and business teams to deliver reliable, high-quality data solutions.
What you’ll be doing
- Designing and developing scalable, testable data pipelines using Python and Apache Spark
- Orchestrating data workflows with AWS tools like Glue, EMR Serverless, Lambda, and S3
- Applying modern software engineering practices: version control, CI/CD, modular design, and automated testing
- Contributing to the development of a lakehouse architecture using Apache Iceberg
- Collaborating with business teams to translate requirements into data-driven solutions
- Building observability into data flows and implementing basic quality checks
- Participating in code reviews, pair programming, and architecture discussions
- Continuously learning about the financial indices domain and sharing insights with the team
What you’ll bring
- Writes clean, maintainable, testable, extensible Python code (ideally with type hints, linters, and tests like pytest)
- Understands data engineering basics: batch processing, schema evolution, and building ETL pipelines
- Has experience with or is eager to learn Apache Spark for large-scale data processing
- Is familiar with the AWS data stack (e.g. S3, Glue, Lambda, EMR)
- Strong experience in relational databases (e.g. Aurora PostgreSQL)
- Enjoys learning the business context and working closely with stakeholders
- Works well in Agile teams and values collaboration over solo heroics
Nice to have
- Experience with Apache Iceberg or similar table formats
- Familiarity with CI/CD tools like GitLab CI, Jenkins, or GitHub Actions
- Exposure to data quality frameworks like Great Expectations or Deequ
- Experience with other data platforms like Databricks
- Curiosity about financial markets, index data, or investment analytics
Join us and be part of a team that values innovation, quality, and continuous improvement. If you’re ready to take your career to the next level and make a significant impact, we’d love to hear from you.
Equal Opportunity and Compliance
We are an equal opportunities employer. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, gender identity or expression, age, marital status, veteran status, pregnancy or disability, or any other basis protected by law. We may accommodate applicants’ religious practices and beliefs, as well as mental health or physical disability needs as required by law.
We are committed to sustainability, and we partner with our customers to help them meet their sustainability objectives. Our charity programs involve volunteering and fundraising opportunities.
Benefits may include healthcare, retirement planning, paid volunteering days and wellbeing initiatives.
Please note: This description may include references to privacy notices and data handling practices. If you are submitting as a Recruitment Agency Partner, please ensure candidates are aware of these notices.
#J-18808-Ljbffr…