The Role
You will be part of a specialist engineering team responsible for designing, building, and optimising end-to-end financial instrument mastering pipelines. These pipelines span ingestion, normalisation, bi-temporal processing, and publication into enterprise data platforms.
You will work closely with data architects, domain experts, and QC engineers to deliver scalable, reliable, and high-performance data solutions across Azure and Microsoft Fabric ecosystems.
Key Responsibilities
- Build and maintain PySpark-based data pipelines for financial instrument mastering across multiple data sources
- Design and implement bi-temporal data processing models (system time + valid time) including Slice, Resolve, Coalesce, and Diff logic
- Develop optimised Azure Cosmos DB data models, including partitioning, indexing, change feed processing, and point-read optimisation
- Integrate external APIs for entity resolution and matching services (PermID / IAAS) with robust retry and batching mechanisms
- Design publication pipelines to convert bi-temporal data into uni-temporal outputs and publish via Microsoft Fabric / Parquet-based lakehouse architectures
- Implement data quality frameworks using Great Expectations to ensure accuracy and compliance
- Build robust unit and integration tests using PyTest for PySpark and Cosmos DB components
- Support and maintain CI/CD pipelines (GitLab CI) including Python packaging, Artifactory deployment, and ARM-based infrastructure provisioning
- Work with YAML-driven configuration for mastering rules, schemas, and environment setup
- Monitor and troubleshoot production pipelines using Eventstream telemetry, KQL, and DataDog observability tools
- Deliver scalable transformation logic, optimised aggregations, and high-performance data processing workflows
- Implement data governance controls including data masking, role-based access, and compliance policies
- Continuously tune and optimise workloads for performance, cost efficiency, and reliability
Required Skills & Experience
- Strong experience in Python and PySpark (Spark SQL, DataFrame API, Structured Streaming)
- Hands-on experience building large-scale ETL / streaming data pipelines
- Experience working with Azure Cosmos DB (NoSQL) including data modelling and performance tuning
- Strong knowledge of Azure Data Lake Storage (ADLS / OneLake / ABFS)
- Experience implementing bi-temporal or SCD Type 2 data models
- Strong understanding of data quality frameworks (e.g., Great Expectations)
- Experience with CI/CD pipelines (GitLab / Azure DevOps) and automated deployments
- Strong testing discipline using PyTest, mocking, and integration testing approaches
- Experience working with YAML/JSON configuration and infrastructure-as-code (ARM templates)
- Strong understanding of distributed data processing and Spark-based architectures
- Experience working with financial or time-series datasets (market data, reference data, risk data preferred)
- Strong communication skills and ability to work with cross-functional stakeholders
Desirable Experience
- Microsoft Fabric (Notebooks, Eventstream, Lakehouses, Spark Job Definitions)
- Financial instrument/reference data (ISIN, CUSIP, LEI, PermID)
- Entity resolution / matching systems and enrichment APIs
- Delta Lake and Change Data Feed (CDF)
- Cosmos DB performance optimisation (RU tuning, bulk operations, concurrency)
- Jinja2 templating or code generation approaches
- SonarQube or similar code quality tooling
- Monorepo development with modern Python packaging tools (uv / Hatchling)
- Power BI / semantic modelling experience
- Knowledge of financial compliance standards (GDPR, SOX)
Technology Stack
Python 3.11+, PySpark 3.5, Spark SQL
Azure Cosmos DB, ADLS, OneLake, Delta Lake, Parquet
Microsoft Fabric (Eventstream, Notebooks, Lakehouse)
Great Expectations, LSEG Data Validation frameworks
GitLab CI/CD, JFrog Artifactory, ARM Templates
DataDog, Eventstream, KQL monitoring
Azure Key Vault, Azure CLI, Fabric APIs
Why Join
- Work on a global financial markets transformation programme
- Hands-on with next-generation Azure + Fabric data platforms
- Exposure to bi-temporal modelling and financial instrument mastering systems
- High-impact engineering role with modern cloud and streaming architecture
- Opportunity to work with leading domain and technical experts in a regulated environment
…