Lead Cloud Data Engineer

Job Description: We are looking for an accomplished Lead Data Engineer to drive the architectural design, development, and optimization of our enterprise-scale data ecosystem. In this senior role, you will spearhead the build-out of high ‑ performance data pipelines, enforce rigorous data quality and validation frameworks, and ensure end ‑ to ‑ end reliability, scalability, and integrity of data flows across the organization. You will play a pivotal role in shaping our data engineering strategy, enabling advanced analytics and mission ‑ critical, data ‑ driven decision-making within our insurance-focused business domains.

Responsibilities: Lead the design and implementation of scalable data architectures on Azure, leveraging Databricks, Delta Lake, and related Azure data services.

Build and optimize high ‑ volume ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, and Delta Live Tables.

Define and enforce best practices for data engineering across notebooks, jobs, CI/CD, Unity Catalog, security, and workspace governance.

Integrate and orchestrate data pipelines using Azure Data Factory, Azure Synapse pipelines, or Azure Databricks Workflows.

Drive performance tuning for Spark jobs, cluster configurations, and data storage layers to balance speed and cost efficiency.

Implement robust data quality and validation frameworks using tools like Great Expectations, DLT expectations, or custom PySpark checks.

Ensure compliance, governance, and lineage tracking through Unity Catalog, Purview integration, and RBAC/ABAC policies.

Architect end ‑ to ‑ end data solutions supporting analytics, ML, actuarial models, and business-critical reporting workloads.

Evaluate new Databricks features, MLflow enhancements, Photon execution, serverless compute, and recommend adoption strategies.

Familiar with working on Agile methodologies - scrum, sprint planning, backlog refinement etc.

Qualifications: 8-12 years experience on Data Engineering role working with Databricks & Azure Cloud technologies.

Bachelor’s degree in computer science, Information Technology, or related field.

Strong proficiency in PySpark, Python, SQL.

Strong experience in data modeling, ETL/ELT pipeline development, and automation

Hands-on experience with performance tuning of data pipelines and workflows

Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc.

Experience with data modeling, ETL processes, Delta Lake and data warehousing.

Experience on Delta Live Tables, Autoloader & Unity Catalog.

Preferred - Knowledge of the insurance industry and its data requirements.

Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.

Excellent communication and problem-solving skills to work effectively with diverse teams

Excellent problem-solving skills and ability to work under tight deadlines.