Sr Cloud Data Engineer

Job Description: We are seeking a highly skilled Senior Data Engineer to lead the design, development, and optimization of scalable data solutions that power enterprise analytics and business decision-making. The ideal candidate will bring deep expertise in Azure and Databricks, strong hands-on experience in building robust ETL/ELT pipelines, and a solid understanding of data quality, governance, and performance tuning. This role will be instrumental in enabling reliable, high-quality data flows across complex systems, with a strong focus on supporting insurance domain use cases, regulatory needs, and data-driven business transformation.

Responsibilities: Lead the design, development, and optimization of scalable, high-performance data pipelines , ensuring reliable data ingestion and transformation across distributed systems, while proactively identifying and resolving performance bottlenecks and system inefficiencies.

Define and enforce enterprise-grade data quality frameworks , implementing advanced validation rules, anomaly detection, and reconciliation processes to ensure data accuracy, completeness, and integrity across large-scale datasets.

Own end-to-end monitoring and reliability of data platforms , establishing observability practices (alerts, SLAs, dashboards) and driving root-cause analysis to minimize downtime and ensure seamless continuity of critical data workflows.

Govern and continuously improve data pipelines through periodic audits and reviews , ensuring adherence to internal standards, data governance policies, and regulatory compliance, while driving standardization and best practices across teams.

Collaborate with cross-functional stakeholders (architects, analysts, product owners) to align data solutions with business objectives, and actively contribute to backlog prioritization, sprint planning, and technical problem-solving in Agile environments.

Mentor junior engineers and drive technical excellence , promoting reusable frameworks, coding standards, and best practices to improve overall team productivity and solution quality.

Qualifications: 6–10 years of hands-on experience in data engineering, with strong expertise in Azure cloud and Databricks-based data platform development.

Bachelor’s degree in computer science, Information Technology, Engineering, or a related discipline.

Strong programming skills in PySpark, Python, and SQL, with proven experience in building scalable and maintainable data pipelines.

Solid experience in data modeling, ETL/ELT pipeline design, Delta Lake, and enterprise data warehousing concepts.

Hands-on expertise with Azure Data Factory, Azure Databricks, Azure Data Lake Storage, and related Azure data services.

Experience with Databricks capabilities such as Delta Live Tables, Auto Loader, Unity Catalog, and workflow orchestration.

Strong understanding of performance tuning, optimization of Spark workloads, and pipeline reliability in large-scale data environments.

Working knowledge of data governance, data quality, and secure data handling practices within enterprise platforms.

Preferred exposure to insurance domain data models, business processes, and regulatory reporting requirements.

Excellent analytical, communication, and problem-solving skills, with the ability to collaborate effectively across cross-functional teams and deliver under tight deadlines.