Lead II - Databricks Engineer

Role description

Lead Databricks Developer (Offshore)

Looking for 2 strong Lead Databricks Developers with streaming expertise.

Key Responsibilities

Design and develop end-to-end data pipelines using Databricks (PySpark, Delta Lake).

Build real-time streaming ingestion pipelines from Kafka and other event-driven sources.

Implement layered data architecture (Bronze, Silver, Gold) to support transformation and reporting.

Develop data processing logic for transactional data ingestion, validation, and enrichment.

Apply data quality, deduplication, and business rules for healthcare datasets.

Support cloud migration from on-premise systems to Databricks + Snowflake ecosystem.

Optimize pipeline performance to meet SLA-driven processing timelines.

Implement and support CI/CD pipelines and DevOps integration.

Collaborate with business and technical stakeholders to deliver scalable solutions.

Required Skills

Strong expertise in Databricks (Streaming & Batch)

Advanced proficiency in PySpark / Spark SQL/Spark Streaming

Experience in real-time data streaming (Kafka)

Strong understanding of data modeling and ETL/ELT pipeline design

Hands-on experience with AWS (S3, Airflow, Lambda)

Experience with CI/CD, Git, and deployment automation

Domain Experience

Experience in Healthcare Payer domain (Claims, Membership, Coverage)

Exposure to FHIR / healthcare data standards preferred

Understanding of data privacy, security, and governance requirements (PII/HIPAA)

Leadership Expectations

Operate at a Lead level driving design, development, and delivery

Mentor team members and ensure delivery quality

Own modules independently and collaborate across distributed teams

Experience

8+ years in Data Engineering

3+ years hands-on experience with Databricks

Prior experience in Tech Lead/Senior roles

Looking for 2 strong Lead Databricks Developers with streaming expertise

Skills

databricks,spark sql,healthcare,apache spark,apache kafka,delta lake,

About UST

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact - touching billions of lives in the process.