Data Engineering Team Lead - Databricks

We are working with a leading global Financial Services business to hire an experienced Data Engineering Team Lead to head up a team of engineers working across a large-scale enterprise data platform. This is a senior hands-on leadership role requiring both deep technical expertise and the ability to drive delivery, mentor talent, and set standards across a distributed engineering function.

Responsibilities:

Lead, mentor, and develop a team of data engineers across multiple locations, driving code reviews, design reviews, and a culture of knowledge-sharing

Own and drive Agile/Scrum delivery processes, ensuring the team operates effectively against roadmap priorities

Design and develop scalable data solutions on a Lakehouse architecture platform, supporting enterprise-wide data processing and analytics

Build, optimise, and maintain ETL/ELT pipelines and structured streaming workflows for both batch and real-time data ingestion

Configure and tune clusters and Spark jobs to deliver consistent performance at scale

Utilise Delta Live Tables and Unity Catalog to manage data ingestion, transformation, and access governance

Apply IAM best practices and uphold compliance with data security and governance standards

Support infrastructure provisioning and resource management using Terraform

Implement monitoring frameworks covering pipeline performance, data quality, and operational health

Contribute to technical documentation and promote continuous improvement across the engineering practice

Requirements

8+ years in data engineering, with at least 3 years hands-on experience with Databricks

Proven experience leading and managing a team of data engineers

Strong Python and Spark programming skills

Solid AWS experience across core services including S3, Glue, and Lambda

Deep understanding of data modelling, SQL, and ETL/ELT design patterns

Experience with Delta Lake, Lakehouse architecture, and Git-based version control

Demonstrable use of AI tools within a professional development workflow

Strong leadership, communication, and stakeholder management skills

Desirable:

Financial services or fund administration background

Exposure to AI/ML implementation patterns and real-time data processing frameworks

Multi-cloud experience beyond AWS

API development or data governance framework experience

Track record of developing junior and mid-level engineers in a fast-paced environment

Benefits

Salary: to be discussed, depending on experience

Length: Permanent contract