Lead Data Engineer

Job Description: A. Data Engineering & Pipeline Development

Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.

Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.

Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.

Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).

Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.

B. Databricks Platform & Cloud Engineering

Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.

Implement data governance, access control, and lineage using Unity Catalog.

Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).

Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.

C. Data Modeling, Architecture & Governance

Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.

Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.

Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).

D. Leadership & Collaboration

Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.

Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.

Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.

Tools & Environment (Typical)

Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.

Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).

Languages: Python, SQL; Scala (nice to have).

DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).

Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.

Responsibilities: A. Data Engineering & Pipeline Development

Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.

Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.

Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.

Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).

Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.

B. Databricks Platform & Cloud Engineering

Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.

Implement data governance, access control, and lineage using Unity Catalog.

Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).

Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.

C. Data Modeling, Architecture & Governance

Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.

Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.

Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).

D. Leadership & Collaboration

Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.

Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.

Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.

Tools & Environment (Typical)

Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.

Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).

Languages: Python, SQL; Scala (nice to have).

DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).

Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.

Qualifications: A. Data Engineering & Pipeline Development

Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.

Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.

Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.

Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).

Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.

B. Databricks Platform & Cloud Engineering

Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.

Implement data governance, access control, and lineage using Unity Catalog.

Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).

Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.

C. Data Modeling, Architecture & Governance

Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.

Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.

Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).

D. Leadership & Collaboration

Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.

Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.

Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.

Tools & Environment (Typical)

Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.

Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).

Languages: Python, SQL; Scala (nice to have).

DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).

Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.