Senior Data Engineer

We are working with a leading global Financial Services business to hire a Senior Data Engineer into their growing data engineering practice. This is a hands-on technical role sitting at the centre of a major enterprise data platform build, with scope to influence architecture, drive pipeline development, and contribute to AI initiatives across a complex, high-volume financial data environment.

Responsibilities:

Design and develop scalable data solutions on a Lakehouse architecture platform, supporting enterprise-wide data processing and analytics

Build, optimise, and maintain ETL/ELT pipelines and structured streaming workflows for both batch and real-time data ingestion

Configure and tune clusters and Spark jobs to deliver consistent performance at scale

Utilise Delta Live Tables and Unity Catalog to manage data ingestion, transformation, and access governance

Apply IAM best practices and maintain compliance with data security standards across the platform

Support infrastructure provisioning and resource management using Terraform

Implement monitoring frameworks covering pipeline performance, data quality, and operational health

Contribute to code reviews, technical documentation, and team knowledge-sharing

Work within an Agile delivery model, collaborating closely with data scientists, analysts, and business stakeholders

Explore emerging technologies and AI tooling to enhance development productivity and platform capability

Requirements

6+ years in data engineering, with at least 2 years hands-on experience with Databricks

Strong Python and Spark programming skills

Solid AWS experience across core services including S3, Glue, and Lambda

Deep understanding of data modelling, SQL, and ETL/ELT design patterns

Experience with Delta Lake, Lakehouse architecture, and Git-based version control

Demonstrable use of AI tools within a professional development workflow

Strong communication skills and the ability to work effectively across technical and non-technical teams

Desirable:

Financial services or fund administration background

Exposure to AI/ML implementation patterns and real-time data processing frameworks

Multi-cloud experience beyond AWS

API development or data governance framework experience

Experience mentoring junior engineers

Benefits

Salary: to be discussed, depending on experience

Length: Permanent contract